Max level shown:
12025-04-30T06:29:41.637ZDEBGtest_image_promotion (init_with_steps): Running step start_crdb
22025-04-30T06:29:41.637ZDEBGtest_image_promotion: Starting CRDB
32025-04-30T06:29:41.710ZINFOtest_image_promotion: cockroach temporary directory: /var/tmp/omicron_tmp/.tmp2lNwUK
42025-04-30T06:29:41.710ZINFOtest_image_promotion: cockroach: copying from seed tarball (/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar) to storage directory (/var/tmp/omicron_tmp/.tmp2lNwUK/data)
52025-04-30T06:29:41.733ZINFOtest_image_promotion: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/var/tmp/omicron_tmp/.tmp2lNwUK/data,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /var/tmp/omicron_tmp/.tmp2lNwUK/listen-url
62025-04-30T06:29:41.734ZINFOtest_image_promotion: cockroach environment: BUILDOMAT_JOB_ID=01JT2JWKGHF3JHFMW0JR13NSA6 BUILDOMAT_TASK_ID=4 CARGO=/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/build/.cargo CARGO_INCREMENTAL=0 CARGO_MANIFEST_DIR=/work/oxidecomputer/omicron/nexus CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=omicron-nexus CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CARGO_TERM_COLOR=always CI=true CRDB_SEED_TAR=/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar GITHUB_REPOSITORY=oxidecomputer/omicron GITHUB_SHA=531912b3fc900900bc7bd18f5eac1ce1e22cb1cd GOTRACEBACK=crash HOME=/home/build LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib LOGNAME=build NEXTEST=1 NEXTEST_BIN_EXE_nexus=/work/oxidecomputer/omicron/target/debug/nexus NEXTEST_BIN_EXE_schema-updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_EXECUTION_MODE=process-per-test NEXTEST_LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib NEXTEST_PROFILE=ci NEXTEST_RUN_ID=29735cc2-0883-4059-8ca2-1da53f01ede3 NEXTEST_TEST_GLOBAL_SLOT=7 NEXTEST_TEST_GROUP=@global NEXTEST_TEST_GROUP_SLOT=none OUT_DIR=/work/oxidecomputer/omicron/target/debug/build/omicron-nexus-553062f859dbbfa0/out PATH=/work/oxidecomputer/omicron/out/mgd/root/opt/oxide/mgd/bin:/work/oxidecomputer/omicron/out/dendrite-stub/bin:/work/oxidecomputer/omicron/out/clickhouse:/work/oxidecomputer/omicron/out/cockroachdb/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin PWD=/work/oxidecomputer/omicron RUSTC_BOOTSTRAP=1 RUSTDOCFLAGS=--document-private-items -D warnings RUSTFLAGS=-D warnings -C link-arg=-R/usr/platform/oxide/lib/amd64 RUSTUP_HOME=/home/build/.rustup RUSTUP_TOOLCHAIN=1.86.0-x86_64-unknown-illumos RUST_BACKTRACE=1 RUST_RECURSION_COUNT=1 SHLVL=1 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TMPDIR=/var/tmp/omicron_tmp TZ=UTC USER=build _=/usr/bin/ptime __NEXTEST_ATTEMPT=1
72025-04-30T06:29:43.108ZINFOtest_image_promotion: cockroach pid: 22831
82025-04-30T06:29:43.108ZINFOtest_image_promotion: cockroach listen URL: postgresql://root@[::1]:45371/omicron?sslmode=disable
92025-04-30T06:29:43.108ZDEBGtest_image_promotion (init_with_steps): Running step start_clickhouse
102025-04-30T06:29:43.109ZDEBGtest_image_promotion: Starting Clickhouse
112025-04-30T06:29:43.870ZDEBGtest_image_promotion (init_with_steps): Running step start_gateway_switch0
122025-04-30T06:29:43.870ZDEBGtest_image_promotion: Starting Management Gateway
132025-04-30T06:29:43.992ZDEBGtest_image_promotion (init_with_steps): Running step start_dendrite_switch0
142025-04-30T06:29:43.992ZDEBGtest_image_promotion: Starting Dendrite for switch0
152025-04-30T06:29:44.030ZDEBGtest_image_promotion (init_with_steps): Running step start_mgd_switch0
162025-04-30T06:29:44.030ZDEBGtest_image_promotion: Starting mgd for switch0
172025-04-30T06:29:44.110ZDEBGtest_image_promotion: mgd port is 40166
182025-04-30T06:29:44.110ZDEBGtest_image_promotion (init_with_steps): Running step record_switch_dns
192025-04-30T06:29:44.110ZDEBGtest_image_promotion: Recording DNS for the switch zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 switch_location = switch0
202025-04-30T06:29:44.110ZDEBGtest_image_promotion (init_with_steps): Running step start_internal_dns
212025-04-30T06:29:44.111ZINFOtest_image_promotion (store): opening sled database path = /var/tmp/omicron_tmp/.tmpknSlPH
222025-04-30T06:29:44.221ZINFOtest_image_promotion (store): pruning trees for generations newer than 0
232025-04-30T06:29:44.221ZINFOtest_image_promotion (store): pruning trees for generations older than 0 keep = 3
242025-04-30T06:29:44.221ZINFOtest_image_promotion (dns): DNS server bound to address kind = dns local_address = [::1]:50517
252025-04-30T06:29:44.222ZDEBGtest_image_promotion (http): registered endpoint kind = dns local_addr = [::1]:37283 method = GET path = /config versions = all
262025-04-30T06:29:44.222ZDEBGtest_image_promotion (http): registered endpoint kind = dns local_addr = [::1]:37283 method = PUT path = /config versions = all
272025-04-30T06:29:44.222ZINFOtest_image_promotion (http): listening kind = dns local_addr = [::1]:37283
282025-04-30T06:29:44.224ZDEBGtest_image_promotion (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:37283
292025-04-30T06:29:44.224ZDEBGtest_image_promotion (init_with_steps): Running step start_external_dns
302025-04-30T06:29:44.224ZINFOtest_image_promotion (store): opening sled database path = /var/tmp/omicron_tmp/.tmpL3wndg
312025-04-30T06:29:44.235ZINFOtest_image_promotion (store): pruning trees for generations newer than 0
322025-04-30T06:29:44.260ZINFOtest_image_promotion (store): pruning trees for generations older than 0 keep = 3
332025-04-30T06:29:44.260ZINFOtest_image_promotion (dns): DNS server bound to address kind = dns local_address = [::1]:35150
342025-04-30T06:29:44.260ZDEBGtest_image_promotion (http): registered endpoint kind = dns local_addr = [::1]:38208 method = GET path = /config versions = all
352025-04-30T06:29:44.260ZDEBGtest_image_promotion (http): registered endpoint kind = dns local_addr = [::1]:38208 method = PUT path = /config versions = all
362025-04-30T06:29:44.260ZINFOtest_image_promotion (http): listening kind = dns local_addr = [::1]:38208
372025-04-30T06:29:44.261ZDEBGtest_image_promotion (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:38208
382025-04-30T06:29:44.262ZDEBGtest_image_promotion (init_with_steps): Running step start_nexus_internal
392025-04-30T06:29:44.262ZDEBGtest_image_promotion: Starting Nexus (internal API)
402025-04-30T06:29:44.263ZINFOtest_image_promotion: setting up nexus server
412025-04-30T06:29:44.266ZINFOtest_image_promotion (ServerContext): registering Oso class class = Action
422025-04-30T06:29:44.266ZINFOtest_image_promotion (ServerContext): registering Oso class class = AnyActor
432025-04-30T06:29:44.266ZINFOtest_image_promotion (ServerContext): registering Oso class class = AuthenticatedActor
442025-04-30T06:29:44.266ZINFOtest_image_promotion (ServerContext): registering Oso class class = BlueprintConfig
452025-04-30T06:29:44.266ZINFOtest_image_promotion (ServerContext): registering Oso class class = Database
462025-04-30T06:29:44.266ZINFOtest_image_promotion (ServerContext): registering Oso class class = DnsConfig
472025-04-30T06:29:44.267ZINFOtest_image_promotion (ServerContext): registering Oso class class = Fleet
482025-04-30T06:29:44.267ZINFOtest_image_promotion (ServerContext): registering Oso class class = Inventory
492025-04-30T06:29:44.267ZINFOtest_image_promotion (ServerContext): registering Oso class class = IpPoolList
502025-04-30T06:29:44.267ZINFOtest_image_promotion (ServerContext): registering Oso class class = ConsoleSessionList
512025-04-30T06:29:44.267ZINFOtest_image_promotion (ServerContext): registering Oso class class = DeviceAuthRequestList
522025-04-30T06:29:44.267ZINFOtest_image_promotion (ServerContext): registering Oso class class = SiloCertificateList
532025-04-30T06:29:44.267ZINFOtest_image_promotion (ServerContext): registering Oso class class = SiloIdentityProviderList
542025-04-30T06:29:44.267ZINFOtest_image_promotion (ServerContext): registering Oso class class = SiloUserList
552025-04-30T06:29:44.268ZINFOtest_image_promotion (ServerContext): registering Oso class class = TargetReleaseConfig
562025-04-30T06:29:44.268ZINFOtest_image_promotion (ServerContext): registering Oso class class = WebhookEventClassList
572025-04-30T06:29:44.268ZINFOtest_image_promotion (ServerContext): registering Oso class class = Project
582025-04-30T06:29:44.268ZINFOtest_image_promotion (ServerContext): registering Oso class class = Disk
592025-04-30T06:29:44.268ZINFOtest_image_promotion (ServerContext): registering Oso class class = Snapshot
602025-04-30T06:29:44.268ZINFOtest_image_promotion (ServerContext): registering Oso class class = ProjectImage
612025-04-30T06:29:44.269ZINFOtest_image_promotion (ServerContext): registering Oso class class = AffinityGroup
622025-04-30T06:29:44.269ZINFOtest_image_promotion (ServerContext): registering Oso class class = AntiAffinityGroup
632025-04-30T06:29:44.269ZINFOtest_image_promotion (ServerContext): registering Oso class class = Instance
642025-04-30T06:29:44.269ZINFOtest_image_promotion (ServerContext): registering Oso class class = IpPool
652025-04-30T06:29:44.269ZINFOtest_image_promotion (ServerContext): registering Oso class class = InstanceNetworkInterface
662025-04-30T06:29:44.269ZINFOtest_image_promotion (ServerContext): registering Oso class class = Vpc
672025-04-30T06:29:44.269ZINFOtest_image_promotion (ServerContext): registering Oso class class = VpcRouter
682025-04-30T06:29:44.269ZINFOtest_image_promotion (ServerContext): registering Oso class class = InternetGateway
692025-04-30T06:29:44.270ZINFOtest_image_promotion (ServerContext): registering Oso class class = InternetGatewayIpPool
702025-04-30T06:29:44.270ZINFOtest_image_promotion (ServerContext): registering Oso class class = InternetGatewayIpAddress
712025-04-30T06:29:44.270ZINFOtest_image_promotion (ServerContext): registering Oso class class = RouterRoute
722025-04-30T06:29:44.270ZINFOtest_image_promotion (ServerContext): registering Oso class class = VpcSubnet
732025-04-30T06:29:44.270ZINFOtest_image_promotion (ServerContext): registering Oso class class = FloatingIp
742025-04-30T06:29:44.270ZINFOtest_image_promotion (ServerContext): registering Oso class class = Image
752025-04-30T06:29:44.270ZINFOtest_image_promotion (ServerContext): registering Oso class class = SiloImage
762025-04-30T06:29:44.270ZINFOtest_image_promotion (ServerContext): registering Oso class class = AddressLot
772025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = Blueprint
782025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = LoopbackAddress
792025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = Certificate
802025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = ConsoleSession
812025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = DeviceAuthRequest
822025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = DeviceAccessToken
832025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = PhysicalDisk
842025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = Rack
852025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = RoleBuiltin
862025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = SshKey
872025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = Silo
882025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = SiloUser
892025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = SiloGroup
902025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = SupportBundle
912025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = IdentityProvider
922025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = SamlIdentityProvider
932025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = Sled
942025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = TufRepo
952025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = TufArtifact
962025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = WebhookEvent
972025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = WebhookReceiver
982025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = WebhookSecret
992025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = Zpool
1002025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = Service
1012025-04-30T06:29:44.271ZINFOtest_image_promotion (ServerContext): registering Oso class class = UserBuiltin
1022025-04-30T06:29:44.272ZINFOtest_image_promotion (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
1032025-04-30T06:29:44.299ZINFOtest_image_promotion (ServerContext): Setting up resolver using DNS address: [::1]:50517
1042025-04-30T06:29:44.300ZINFOtest_image_promotion (DnsResolver): new DNS resolver addresses = [[::1]:50517]
1052025-04-30T06:29:44.300ZINFOtest_image_promotion (ServerContext): Setting up qorb database pool from a single host url = PostgresConfigWithUrl {\n url_raw: "postgresql://root@[::1]:45371/omicron?sslmode=disable",\n config: Config {\n user: Some(\n "root",\n ),\n password: None,\n dbname: Some(\n "omicron",\n ),\n options: None,\n application_name: None,\n ssl_mode: Disable,\n host: [\n Tcp(\n "::1",\n ),\n ],\n hostaddr: [],\n port: [\n 45371,\n ],\n connect_timeout: None,\n tcp_user_timeout: None,\n keepalives: true,\n keepalives_idle: 7200s,\n keepalives_interval: None,\n keepalives_retries: None,\n target_session_attrs: Any,\n channel_binding: Prefer,\n load_balance_hosts: Disable,\n },\n}
1062025-04-30T06:29:44.302ZDEBGtest_image_promotion (ServerContext): registered USDT probes
1072025-04-30T06:29:44.391ZINFOtest_image_promotion (nexus): Database schema version is up to date desired_version = 140.0.0 found_version = 140.0.0
1082025-04-30T06:29:44.522ZDEBGtest_image_promotion (clickhouse-client): registered USDT probes id = eaedc193-8679-42a0-9192-cd7432825d42
1092025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /bgtasks versions = all
1102025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /bgtasks/activate versions = all
1112025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /bgtasks/view/{bgtask_name} versions = all
1122025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /clickhouse/policy versions = all
1132025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /clickhouse/policy versions = all
1142025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
1152025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1162025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1172025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1182025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1192025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /demo-saga versions = all
1202025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1212025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /deployment/blueprints/all versions = all
1222025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1232025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1242025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /deployment/blueprints/import versions = all
1252025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /deployment/blueprints/regenerate versions = all
1262025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /deployment/blueprints/target versions = all
1272025-04-30T06:29:44.611ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /deployment/blueprints/target versions = all
1282025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = PUT path = /deployment/blueprints/target/enabled versions = all
1292025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1302025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = PUT path = /disks/{disk_id} versions = all
1312025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /experimental/v1/system/support-bundles versions = all
1322025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /experimental/v1/system/support-bundles versions = all
1332025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1342025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1352025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1362025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1372025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1382025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1392025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
1402025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /instances/{instance_id}/migrate versions = all
1412025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /metrics/collectors versions = all
1422025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1432025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /metrics/producers versions = all
1442025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /mgs-updates versions = all
1452025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1462025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /oximeter/read-policy versions = all
1472025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /oximeter/read-policy versions = all
1482025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /physical-disk/expunge versions = all
1492025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /probes/{sled} versions = all
1502025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1512025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /sagas versions = all
1522025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /sagas/{saga_id} versions = all
1532025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /sled-agents/{sled_id} versions = all
1542025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /sled-agents/{sled_id} versions = all
1552025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1562025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /sleds/add versions = all
1572025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /sleds/expunge versions = all
1582025-04-30T06:29:44.612ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /sleds/uninitialized versions = all
1592025-04-30T06:29:44.613ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = PUT path = /switch/{switch_id} versions = all
1602025-04-30T06:29:44.613ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = GET path = /v1/ping versions = all
1612025-04-30T06:29:44.613ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = PUT path = /vmms/{propolis_id} versions = all
1622025-04-30T06:29:44.613ZDEBGtest_image_promotion (dropshot_internal): registered endpoint local_addr = 127.0.0.1:45630 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1632025-04-30T06:29:44.613ZINFOtest_image_promotion (dropshot_internal): listening local_addr = 127.0.0.1:45630
1642025-04-30T06:29:44.613ZDEBGtest_image_promotion (dropshot_internal): successfully registered DTrace USDT probes local_addr = 127.0.0.1:45630
1652025-04-30T06:29:44.613ZINFOtest_image_promotion (SEC): SEC running sec_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
1662025-04-30T06:29:44.613ZTRACtest_image_promotion (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1672025-04-30T06:29:44.614ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
1682025-04-30T06:29:44.618ZDEBGtest_image_promotion (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1692025-04-30T06:29:44.619ZDEBGtest_image_promotion (DataLoader): attempting to create built-in users
1702025-04-30T06:29:44.621ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1712025-04-30T06:29:44.621ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
1722025-04-30T06:29:44.624ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1732025-04-30T06:29:44.624ZINFOtest_image_promotion (MgsUpdateDriver): starting MgsUpdateDriver
1742025-04-30T06:29:44.631ZTRACtest_image_promotion (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 35 4d 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 5M..........._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:61142 req_id = f23a5caa-c509-4fd4-884a-da0e294066b5
1752025-04-30T06:29:44.632ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13645,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61142 req_id = f23a5caa-c509-4fd4-884a-da0e294066b5
1762025-04-30T06:29:44.634ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13645, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_crucible-pantry._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 16, 95, 99, 114, 117, 99, 105, 98, 108, 101, 45, 112, 97, 110, 116, 114, 121, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_crucible-pantry._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61142 req_id = f23a5caa-c509-4fd4-884a-da0e294066b5
1772025-04-30T06:29:44.697ZINFOtest_image_promotion (DataLoader): created 5 built-in users
1782025-04-30T06:29:44.697ZTRACtest_image_promotion (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1792025-04-30T06:29:44.697ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
1802025-04-30T06:29:44.701ZDEBGtest_image_promotion (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1812025-04-30T06:29:44.701ZDEBGtest_image_promotion (DataLoader): attempting to create built-in roles
1822025-04-30T06:29:44.701ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1832025-04-30T06:29:44.701ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
1842025-04-30T06:29:44.701ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1852025-04-30T06:29:44.717ZINFOtest_image_promotion (DataLoader): created 10 built-in roles
1862025-04-30T06:29:44.717ZTRACtest_image_promotion (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1872025-04-30T06:29:44.717ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
1882025-04-30T06:29:44.724ZDEBGtest_image_promotion (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1892025-04-30T06:29:44.725ZDEBGtest_image_promotion (DataLoader): attempting to create built-in role assignments
1902025-04-30T06:29:44.725ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1912025-04-30T06:29:44.725ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
1922025-04-30T06:29:44.726ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1932025-04-30T06:29:45.045ZINFOtest_image_promotion (DataLoader): created 4 built-in role assignments
1942025-04-30T06:29:45.045ZTRACtest_image_promotion (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1952025-04-30T06:29:45.045ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
1962025-04-30T06:29:45.047ZDEBGtest_image_promotion (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1972025-04-30T06:29:45.048ZDEBGtest_image_promotion (DataLoader): attempting to create built-in silos
1982025-04-30T06:29:45.048ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1992025-04-30T06:29:45.048ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2002025-04-30T06:29:45.048ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2012025-04-30T06:29:45.139ZINFOtest_image_promotion (DataLoader): created 2 built-in silos
2022025-04-30T06:29:45.140ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2032025-04-30T06:29:45.140ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2042025-04-30T06:29:45.140ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2052025-04-30T06:29:45.157ZTRACtest_image_promotion (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2062025-04-30T06:29:45.157ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2072025-04-30T06:29:45.159ZDEBGtest_image_promotion (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2082025-04-30T06:29:45.159ZDEBGtest_image_promotion (DataLoader): attempting to create built-in projects
2092025-04-30T06:29:45.159ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2102025-04-30T06:29:45.159ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2112025-04-30T06:29:45.160ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2122025-04-30T06:29:45.168ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2132025-04-30T06:29:45.168ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2142025-04-30T06:29:45.168ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2152025-04-30T06:29:45.168ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2162025-04-30T06:29:45.169ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2172025-04-30T06:29:45.225ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2182025-04-30T06:29:45.225ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2192025-04-30T06:29:45.226ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2202025-04-30T06:29:45.226ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2212025-04-30T06:29:45.231ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2222025-04-30T06:29:45.235ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2232025-04-30T06:29:45.235ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2242025-04-30T06:29:45.235ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2252025-04-30T06:29:45.235ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2262025-04-30T06:29:45.235ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2272025-04-30T06:29:45.235ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2282025-04-30T06:29:45.237ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2292025-04-30T06:29:45.237ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2302025-04-30T06:29:45.237ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2312025-04-30T06:29:45.237ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2322025-04-30T06:29:45.241ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2332025-04-30T06:29:45.245ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2342025-04-30T06:29:45.245ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2352025-04-30T06:29:45.245ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2362025-04-30T06:29:45.245ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2372025-04-30T06:29:45.267ZINFOtest_image_promotion (DataLoader): created built-in services project
2382025-04-30T06:29:45.267ZTRACtest_image_promotion (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2392025-04-30T06:29:45.267ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2402025-04-30T06:29:45.271ZDEBGtest_image_promotion (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2412025-04-30T06:29:45.271ZDEBGtest_image_promotion (DataLoader): attempting to create built-in VPCs
2422025-04-30T06:29:45.271ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2432025-04-30T06:29:45.271ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2442025-04-30T06:29:45.272ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2452025-04-30T06:29:45.275ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2462025-04-30T06:29:45.275ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2472025-04-30T06:29:45.276ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2482025-04-30T06:29:45.279ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2492025-04-30T06:29:45.279ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2502025-04-30T06:29:45.279ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2512025-04-30T06:29:45.280ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2522025-04-30T06:29:45.280ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2532025-04-30T06:29:45.289ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2542025-04-30T06:29:45.289ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2552025-04-30T06:29:45.289ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2562025-04-30T06:29:45.292ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-04-30T06:29:45.296ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2582025-04-30T06:29:45.298ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2592025-04-30T06:29:45.298ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2602025-04-30T06:29:45.298ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2612025-04-30T06:29:45.300ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2622025-04-30T06:29:45.306ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2632025-04-30T06:29:45.307ZTRACtest_image_promotion (DataLoader): successful lookup of siloed resource "Project" using built-in user
2642025-04-30T06:29:45.307ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2652025-04-30T06:29:45.307ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2662025-04-30T06:29:45.307ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2672025-04-30T06:29:45.307ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2682025-04-30T06:29:45.307ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2692025-04-30T06:29:45.311ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2702025-04-30T06:29:45.311ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2712025-04-30T06:29:45.311ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2722025-04-30T06:29:45.312ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2732025-04-30T06:29:45.314ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2742025-04-30T06:29:45.315ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2752025-04-30T06:29:45.315ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2762025-04-30T06:29:45.316ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2772025-04-30T06:29:45.325ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2782025-04-30T06:29:45.332ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2792025-04-30T06:29:45.332ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2802025-04-30T06:29:45.333ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2812025-04-30T06:29:45.334ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2822025-04-30T06:29:45.351ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
2832025-04-30T06:29:45.351ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2842025-04-30T06:29:45.351ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2852025-04-30T06:29:45.351ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2862025-04-30T06:29:45.352ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2872025-04-30T06:29:45.374ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2882025-04-30T06:29:45.374ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2892025-04-30T06:29:45.374ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2902025-04-30T06:29:45.375ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2912025-04-30T06:29:45.381ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2922025-04-30T06:29:45.381ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2932025-04-30T06:29:45.381ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2942025-04-30T06:29:45.383ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2952025-04-30T06:29:45.387ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2962025-04-30T06:29:45.396ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
2972025-04-30T06:29:45.397ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2982025-04-30T06:29:45.397ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
2992025-04-30T06:29:45.398ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3002025-04-30T06:29:45.406ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3012025-04-30T06:29:45.406ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3022025-04-30T06:29:45.407ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3032025-04-30T06:29:45.419ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3042025-04-30T06:29:45.419ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3052025-04-30T06:29:45.420ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3062025-04-30T06:29:45.440ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3072025-04-30T06:29:45.440ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3082025-04-30T06:29:45.440ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3092025-04-30T06:29:45.440ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3102025-04-30T06:29:45.441ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3112025-04-30T06:29:45.446ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3122025-04-30T06:29:45.446ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3132025-04-30T06:29:45.446ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3142025-04-30T06:29:45.446ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-04-30T06:29:45.461ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3162025-04-30T06:29:45.461ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3172025-04-30T06:29:45.461ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3182025-04-30T06:29:45.462ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3192025-04-30T06:29:45.474ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3202025-04-30T06:29:45.479ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3212025-04-30T06:29:45.480ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3222025-04-30T06:29:45.481ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3232025-04-30T06:29:45.481ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3242025-04-30T06:29:45.590ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3252025-04-30T06:29:45.590ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3262025-04-30T06:29:45.590ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3272025-04-30T06:29:45.590ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3282025-04-30T06:29:45.591ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3292025-04-30T06:29:45.593ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3302025-04-30T06:29:45.593ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3312025-04-30T06:29:45.593ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3322025-04-30T06:29:45.594ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-04-30T06:29:45.595ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3342025-04-30T06:29:45.595ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3352025-04-30T06:29:45.595ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3362025-04-30T06:29:45.596ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3372025-04-30T06:29:45.599ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3382025-04-30T06:29:45.605ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3392025-04-30T06:29:45.605ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3402025-04-30T06:29:45.605ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3412025-04-30T06:29:45.605ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3422025-04-30T06:29:45.674ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3432025-04-30T06:29:45.674ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3442025-04-30T06:29:45.674ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3452025-04-30T06:29:45.674ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3462025-04-30T06:29:45.675ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3472025-04-30T06:29:45.677ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3482025-04-30T06:29:45.677ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3492025-04-30T06:29:45.677ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3502025-04-30T06:29:45.678ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-04-30T06:29:45.680ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3522025-04-30T06:29:45.680ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3532025-04-30T06:29:45.680ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3542025-04-30T06:29:45.680ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3552025-04-30T06:29:45.683ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3562025-04-30T06:29:45.688ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3572025-04-30T06:29:45.688ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3582025-04-30T06:29:45.688ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3592025-04-30T06:29:45.688ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3602025-04-30T06:29:45.693ZDEBGtest_image_promotion (DataLoader): attempting to create built-in VPC firewall rules
3612025-04-30T06:29:45.694ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3622025-04-30T06:29:45.694ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3632025-04-30T06:29:45.694ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3642025-04-30T06:29:45.697ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3652025-04-30T06:29:45.697ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3662025-04-30T06:29:45.698ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3672025-04-30T06:29:45.700ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3682025-04-30T06:29:45.700ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3692025-04-30T06:29:45.700ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3702025-04-30T06:29:45.703ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3712025-04-30T06:29:45.704ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3722025-04-30T06:29:45.704ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3732025-04-30T06:29:45.704ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3742025-04-30T06:29:45.704ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3752025-04-30T06:29:45.705ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3762025-04-30T06:29:45.705ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3772025-04-30T06:29:45.705ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3782025-04-30T06:29:45.705ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3792025-04-30T06:29:45.707ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3802025-04-30T06:29:45.707ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3812025-04-30T06:29:45.707ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3822025-04-30T06:29:45.707ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3832025-04-30T06:29:45.709ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3842025-04-30T06:29:45.714ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3852025-04-30T06:29:45.714ZTRACtest_image_promotion (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
3862025-04-30T06:29:45.714ZTRACtest_image_promotion (DataLoader): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3872025-04-30T06:29:45.715ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3882025-04-30T06:29:45.715ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3892025-04-30T06:29:45.715ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3902025-04-30T06:29:45.715ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3912025-04-30T06:29:45.717ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3922025-04-30T06:29:45.717ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3932025-04-30T06:29:45.717ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3942025-04-30T06:29:45.717ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3952025-04-30T06:29:45.719ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3962025-04-30T06:29:45.719ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3972025-04-30T06:29:45.720ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
3982025-04-30T06:29:45.720ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3992025-04-30T06:29:45.722ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4002025-04-30T06:29:45.728ZDEBGtest_image_promotion (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4012025-04-30T06:29:45.728ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4022025-04-30T06:29:45.728ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4032025-04-30T06:29:45.729ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4042025-04-30T06:29:45.738ZTRACtest_image_promotion (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4052025-04-30T06:29:45.738ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4062025-04-30T06:29:45.738ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4072025-04-30T06:29:45.738ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4082025-04-30T06:29:45.738ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4092025-04-30T06:29:45.740ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4102025-04-30T06:29:45.740ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4112025-04-30T06:29:45.740ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4122025-04-30T06:29:45.741ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4132025-04-30T06:29:45.743ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4142025-04-30T06:29:45.743ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4152025-04-30T06:29:45.743ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4162025-04-30T06:29:45.744ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4172025-04-30T06:29:45.749ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4182025-04-30T06:29:45.757ZDEBGtest_image_promotion (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4192025-04-30T06:29:45.758ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4202025-04-30T06:29:45.758ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4212025-04-30T06:29:45.758ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4222025-04-30T06:29:45.837ZINFOtest_image_promotion (DataLoader): created built-in VPC firewall rules
4232025-04-30T06:29:45.837ZDEBGtest_image_promotion (DataLoader): attempting to create built-in VPC Subnets
4242025-04-30T06:29:45.837ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4252025-04-30T06:29:45.837ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4262025-04-30T06:29:45.838ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4272025-04-30T06:29:45.843ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4282025-04-30T06:29:45.843ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4292025-04-30T06:29:45.844ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4302025-04-30T06:29:45.847ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4312025-04-30T06:29:45.847ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4322025-04-30T06:29:45.847ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4332025-04-30T06:29:45.850ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4342025-04-30T06:29:45.850ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4352025-04-30T06:29:45.850ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4362025-04-30T06:29:45.850ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4372025-04-30T06:29:45.850ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4382025-04-30T06:29:45.852ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4392025-04-30T06:29:45.852ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4402025-04-30T06:29:45.852ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4412025-04-30T06:29:45.853ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4422025-04-30T06:29:45.856ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4432025-04-30T06:29:45.856ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4442025-04-30T06:29:45.856ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4452025-04-30T06:29:45.856ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4462025-04-30T06:29:45.858ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4472025-04-30T06:29:45.864ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4482025-04-30T06:29:45.864ZTRACtest_image_promotion (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4492025-04-30T06:29:45.864ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4502025-04-30T06:29:45.864ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4512025-04-30T06:29:45.864ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4522025-04-30T06:29:45.873ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4532025-04-30T06:29:45.873ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4542025-04-30T06:29:45.873ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4552025-04-30T06:29:45.873ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4562025-04-30T06:29:45.874ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4572025-04-30T06:29:45.875ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4582025-04-30T06:29:45.875ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4592025-04-30T06:29:45.875ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4602025-04-30T06:29:45.876ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4612025-04-30T06:29:45.878ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4622025-04-30T06:29:45.878ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4632025-04-30T06:29:45.878ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4642025-04-30T06:29:45.878ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4652025-04-30T06:29:45.881ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4662025-04-30T06:29:45.887ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4672025-04-30T06:29:45.893ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
4682025-04-30T06:29:45.893ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4692025-04-30T06:29:45.893ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4702025-04-30T06:29:45.894ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4712025-04-30T06:29:45.896ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4722025-04-30T06:29:45.900ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4732025-04-30T06:29:45.900ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4742025-04-30T06:29:45.900ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4752025-04-30T06:29:45.901ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4762025-04-30T06:29:45.904ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4772025-04-30T06:29:45.904ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4782025-04-30T06:29:45.904ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4792025-04-30T06:29:45.904ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4802025-04-30T06:29:45.907ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4812025-04-30T06:29:45.915ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
4822025-04-30T06:29:45.915ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4832025-04-30T06:29:45.915ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4842025-04-30T06:29:45.916ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4852025-04-30T06:29:45.973ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4862025-04-30T06:29:45.973ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4872025-04-30T06:29:45.974ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4882025-04-30T06:29:45.990ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4892025-04-30T06:29:45.990ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4902025-04-30T06:29:45.990ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4912025-04-30T06:29:45.990ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4922025-04-30T06:29:45.991ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4932025-04-30T06:29:46.003ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4942025-04-30T06:29:46.003ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4952025-04-30T06:29:46.003ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
4962025-04-30T06:29:46.004ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4972025-04-30T06:29:46.009ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4982025-04-30T06:29:46.009ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4992025-04-30T06:29:46.009ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5002025-04-30T06:29:46.009ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5012025-04-30T06:29:46.019ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5022025-04-30T06:29:46.029ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5032025-04-30T06:29:46.038ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5042025-04-30T06:29:46.038ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5052025-04-30T06:29:46.038ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5062025-04-30T06:29:46.038ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5072025-04-30T06:29:46.039ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5082025-04-30T06:29:46.049ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5092025-04-30T06:29:46.049ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5102025-04-30T06:29:46.049ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5112025-04-30T06:29:46.050ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5122025-04-30T06:29:46.056ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5132025-04-30T06:29:46.056ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5142025-04-30T06:29:46.056ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5152025-04-30T06:29:46.057ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5162025-04-30T06:29:46.060ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5172025-04-30T06:29:46.069ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5182025-04-30T06:29:46.071ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5192025-04-30T06:29:46.071ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5202025-04-30T06:29:46.072ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5212025-04-30T06:29:46.079ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5222025-04-30T06:29:46.079ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5232025-04-30T06:29:46.080ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5242025-04-30T06:29:46.087ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5252025-04-30T06:29:46.087ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5262025-04-30T06:29:46.087ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5272025-04-30T06:29:46.088ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5282025-04-30T06:29:46.088ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5292025-04-30T06:29:46.091ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5302025-04-30T06:29:46.091ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5312025-04-30T06:29:46.091ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5322025-04-30T06:29:46.091ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5332025-04-30T06:29:46.095ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5342025-04-30T06:29:46.095ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5352025-04-30T06:29:46.096ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5362025-04-30T06:29:46.096ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5372025-04-30T06:29:46.099ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5382025-04-30T06:29:46.107ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5392025-04-30T06:29:46.113ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5402025-04-30T06:29:46.113ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5412025-04-30T06:29:46.113ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5422025-04-30T06:29:46.114ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5432025-04-30T06:29:46.114ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5442025-04-30T06:29:46.117ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5452025-04-30T06:29:46.117ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5462025-04-30T06:29:46.117ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5472025-04-30T06:29:46.118ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5482025-04-30T06:29:46.124ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5492025-04-30T06:29:46.124ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5502025-04-30T06:29:46.124ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5512025-04-30T06:29:46.124ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5522025-04-30T06:29:46.129ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5532025-04-30T06:29:46.142ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5542025-04-30T06:29:46.142ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5552025-04-30T06:29:46.142ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5562025-04-30T06:29:46.142ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5572025-04-30T06:29:46.160ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5582025-04-30T06:29:46.161ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5592025-04-30T06:29:46.163ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5602025-04-30T06:29:46.166ZINFOtest_image_promotion (DataLoader): created built-in services vpc subnets
5612025-04-30T06:29:46.166ZINFOtest_image_promotion (DataLoader): created built-in services vpc
5622025-04-30T06:29:46.166ZTRACtest_image_promotion (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5632025-04-30T06:29:46.166ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5642025-04-30T06:29:46.170ZDEBGtest_image_promotion (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5652025-04-30T06:29:46.170ZDEBGtest_image_promotion (DataLoader): attempting to create silo users
5662025-04-30T06:29:46.170ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5672025-04-30T06:29:46.170ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5682025-04-30T06:29:46.171ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5692025-04-30T06:29:46.179ZINFOtest_image_promotion (DataLoader): created 2 silo users
5702025-04-30T06:29:46.179ZDEBGtest_image_promotion (DataLoader): attempting to create silo user role assignments
5712025-04-30T06:29:46.179ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5722025-04-30T06:29:46.179ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5732025-04-30T06:29:46.181ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5742025-04-30T06:29:46.185ZINFOtest_image_promotion (DataLoader): created 2 silo user role assignments
5752025-04-30T06:29:46.185ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5762025-04-30T06:29:46.185ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5772025-04-30T06:29:46.186ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5782025-04-30T06:29:46.194ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5792025-04-30T06:29:46.194ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5802025-04-30T06:29:46.194ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5812025-04-30T06:29:46.204ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5822025-04-30T06:29:46.204ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5832025-04-30T06:29:46.204ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5842025-04-30T06:29:46.204ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5852025-04-30T06:29:46.205ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5862025-04-30T06:29:46.210ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5872025-04-30T06:29:46.212ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5882025-04-30T06:29:46.213ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5892025-04-30T06:29:46.213ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5902025-04-30T06:29:46.213ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5912025-04-30T06:29:46.221ZTRACtest_image_promotion (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5922025-04-30T06:29:46.221ZTRACtest_image_promotion (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5932025-04-30T06:29:46.221ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5942025-04-30T06:29:46.221ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5952025-04-30T06:29:46.222ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5962025-04-30T06:29:46.226ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
5972025-04-30T06:29:46.229ZDEBGtest_image_promotion (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5982025-04-30T06:29:46.229ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5992025-04-30T06:29:46.229ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
6002025-04-30T06:29:46.230ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6012025-04-30T06:29:46.339ZTRACtest_image_promotion (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6022025-04-30T06:29:46.339ZDEBGtest_image_promotion (DataLoader): roles roles = RoleSet { roles: {} }
6032025-04-30T06:29:46.340ZDEBGtest_image_promotion (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6042025-04-30T06:29:46.359ZINFOtest_image_promotion (nexus): populate complete
6052025-04-30T06:29:46.359ZINFOtest_image_promotion (nexus): activating background tasks
6062025-04-30T06:29:46.363ZDEBGtest_image_promotion (init_with_steps): Running step start_sled1
6072025-04-30T06:29:46.363ZINFOtest_image_promotion (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6082025-04-30T06:29:46.419ZINFOtest_image_promotion (SledAgent): created simulated sled agent server = b6d65341-167c-41df-9b5c-41cded99c229 sim_mode = Explicit sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6092025-04-30T06:29:46.420ZINFOtest_image_promotion (omicron_sled_agent::sim::SimulatedUpstairs): registered sled b6d65341-167c-41df-9b5c-41cded99c229 storage
6102025-04-30T06:29:46.420ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
6112025-04-30T06:29:46.420ZDEBGtest_image_promotion (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6122025-04-30T06:29:46.420ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
6132025-04-30T06:29:46.420ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6142025-04-30T06:29:46.421ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6152025-04-30T06:29:46.421ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
6162025-04-30T06:29:46.421ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
6172025-04-30T06:29:46.421ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
6182025-04-30T06:29:46.421ZWARNtest_image_promotion (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config nor servers
6192025-04-30T06:29:46.422ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 111.512\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6202025-04-30T06:29:46.422ZINFOtest_image_promotion (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmp1ODgKk/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6212025-04-30T06:29:46.422ZDEBGtest_image_promotion (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmp1ODgKk/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6222025-04-30T06:29:46.422ZINFOtest_image_promotion (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmp1371pQ/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6232025-04-30T06:29:46.422ZDEBGtest_image_promotion (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmp1371pQ/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6242025-04-30T06:29:46.476ZDEBGtest_image_promotion (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:41286 method = GET path = /artifact/sha256/{sha256} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6252025-04-30T06:29:46.478ZINFOtest_image_promotion (dropshot (Repo Depot)): listening local_addr = [::1]:41286 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6262025-04-30T06:29:46.479ZDEBGtest_image_promotion (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:41286 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6272025-04-30T06:29:46.485ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /artifacts sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6282025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /artifacts/{sha256} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6292025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6302025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6312025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6322025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6332025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6342025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6352025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /bootstore/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6362025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6372025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /disks/{disk_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6382025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = POST path = /disks/{disk_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6392025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /eip-gateways sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6402025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6412025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6422025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6432025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6442025-04-30T06:29:46.486ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /omicron-physical-disks sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6452025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /sled-identifiers sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6462025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /sled-role sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6472025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /sleds sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6482025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support/dladm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6492025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support/ipadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6502025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support/logs/download/{zone} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6512025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support/logs/zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6522025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support/nvmeadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6532025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support/pargs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6542025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support/pfiles-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6552025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support/pstack-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6562025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support/zfs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6572025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support/zoneadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6582025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support/zpool-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6592025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6602025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6612025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6622025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6632025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6642025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6652025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6662025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6672025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6682025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6692025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /timesync sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6702025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = DELETE path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6712025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6722025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6732025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = DELETE path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6742025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6752025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6762025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6772025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6782025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = POST path = /vmms/{propolis_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6792025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6802025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6812025-04-30T06:29:46.487ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6822025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6832025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6842025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6852025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6862025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6872025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = POST path = /zones/bundle-cleanup sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6882025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6892025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = PUT path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6902025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /zones/bundle-cleanup/utilization sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6912025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /zones/bundles sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6922025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /zones/bundles/{zone_name} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6932025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6942025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6952025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:60070 method = GET path = /zpools sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6962025-04-30T06:29:46.488ZINFOtest_image_promotion (dropshot): listening local_addr = [::1]:60070 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6972025-04-30T06:29:46.488ZDEBGtest_image_promotion (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:60070 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6982025-04-30T06:29:46.488ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
6992025-04-30T06:29:46.488ZDEBGtest_image_promotion (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
7002025-04-30T06:29:46.488ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
7012025-04-30T06:29:46.488ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7022025-04-30T06:29:46.488ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7032025-04-30T06:29:46.488ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
7042025-04-30T06:29:46.488ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
7052025-04-30T06:29:46.489ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
7062025-04-30T06:29:46.489ZWARNtest_image_promotion (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config nor servers
7072025-04-30T06:29:46.489ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 56.853\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
7082025-04-30T06:29:46.489ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
7092025-04-30T06:29:46.489ZINFOtest_image_promotion (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-04-30 06:19:46.489048467 UTC
7102025-04-30T06:29:46.489ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database
7112025-04-30T06:29:46.489ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
7122025-04-30T06:29:46.489ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
7132025-04-30T06:29:46.490ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
7142025-04-30T06:29:46.490ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
7152025-04-30T06:29:46.490ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7162025-04-30T06:29:46.490ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
7172025-04-30T06:29:46.490ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7182025-04-30T06:29:46.490ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
7192025-04-30T06:29:46.490ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
7202025-04-30T06:29:46.490ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
7212025-04-30T06:29:46.490ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7222025-04-30T06:29:46.491ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7232025-04-30T06:29:46.491ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
7242025-04-30T06:29:46.491ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
7252025-04-30T06:29:46.491ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
7262025-04-30T06:29:46.491ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
7272025-04-30T06:29:46.491ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
7282025-04-30T06:29:46.491ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
7292025-04-30T06:29:46.491ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig
7302025-04-30T06:29:46.491ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7312025-04-30T06:29:46.492ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
7322025-04-30T06:29:46.492ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7332025-04-30T06:29:46.492ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7342025-04-30T06:29:46.492ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
7352025-04-30T06:29:46.492ZWARNtest_image_promotion (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = no blueprint
7362025-04-30T06:29:46.492ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 64.919\u{b5}s iteration = 1 status = {"error":"no blueprint"}
7372025-04-30T06:29:46.492ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
7382025-04-30T06:29:46.492ZWARNtest_image_promotion (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector reason = no blueprint
7392025-04-30T06:29:46.492ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 16.232\u{b5}s iteration = 1 status = {"error":"no blueprint"}
7402025-04-30T06:29:46.492ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
7412025-04-30T06:29:46.492ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
7422025-04-30T06:29:46.492ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7432025-04-30T06:29:46.492ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
7442025-04-30T06:29:46.492ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7452025-04-30T06:29:46.493ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7462025-04-30T06:29:46.493ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
7472025-04-30T06:29:46.493ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
7482025-04-30T06:29:46.493ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7492025-04-30T06:29:46.493ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
7502025-04-30T06:29:46.493ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
7512025-04-30T06:29:46.493ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
7522025-04-30T06:29:46.494ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
7532025-04-30T06:29:46.494ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 19.188\u{b5}s iteration = 1 status = {"error":"task disabled"}
7542025-04-30T06:29:46.494ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
7552025-04-30T06:29:46.494ZWARNtest_image_promotion (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
7562025-04-30T06:29:46.494ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 16.182\u{b5}s iteration = 1 status = {"error":"no blueprint"}
7572025-04-30T06:29:46.494ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
7582025-04-30T06:29:46.494ZINFOtest_image_promotion (BackgroundTasks): Decommissioned Disk Cleaner disabled background_task = decommissioned_disk_cleaner
7592025-04-30T06:29:46.494ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 75.53\u{b5}s iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
7602025-04-30T06:29:46.494ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
7612025-04-30T06:29:46.494ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
7622025-04-30T06:29:46.494ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7632025-04-30T06:29:46.494ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
7642025-04-30T06:29:46.494ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7652025-04-30T06:29:46.494ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7662025-04-30T06:29:46.494ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
7672025-04-30T06:29:46.494ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
7682025-04-30T06:29:46.494ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7692025-04-30T06:29:46.494ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
7702025-04-30T06:29:46.495ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7712025-04-30T06:29:46.495ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7722025-04-30T06:29:46.495ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
7732025-04-30T06:29:46.495ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
7742025-04-30T06:29:46.495ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7752025-04-30T06:29:46.498ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7762025-04-30T06:29:46.498ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
7772025-04-30T06:29:46.498ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
7782025-04-30T06:29:46.498ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7792025-04-30T06:29:46.499ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7802025-04-30T06:29:46.499ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
7812025-04-30T06:29:46.499ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
7822025-04-30T06:29:46.499ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7832025-04-30T06:29:46.499ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7842025-04-30T06:29:46.499ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
7852025-04-30T06:29:46.499ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
7862025-04-30T06:29:46.500ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7872025-04-30T06:29:46.500ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
7882025-04-30T06:29:46.500ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7892025-04-30T06:29:46.500ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7902025-04-30T06:29:46.500ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
7912025-04-30T06:29:46.500ZINFOtest_image_promotion (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
7922025-04-30T06:29:46.500ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = instance_updater elapsed = 296.791\u{b5}s iteration = 1 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7932025-04-30T06:29:46.500ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
7942025-04-30T06:29:46.500ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
7952025-04-30T06:29:46.501ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7962025-04-30T06:29:46.501ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7972025-04-30T06:29:46.501ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
7982025-04-30T06:29:46.501ZDEBGtest_image_promotion (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7992025-04-30T06:29:46.501ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
8002025-04-30T06:29:46.501ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8012025-04-30T06:29:46.502ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8022025-04-30T06:29:46.502ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
8032025-04-30T06:29:46.502ZINFOtest_image_promotion (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
8042025-04-30T06:29:46.502ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
8052025-04-30T06:29:46.502ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8062025-04-30T06:29:46.502ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
8072025-04-30T06:29:46.502ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8082025-04-30T06:29:46.503ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8092025-04-30T06:29:46.503ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
8102025-04-30T06:29:46.503ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database
8112025-04-30T06:29:46.503ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
8122025-04-30T06:29:46.504ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
8132025-04-30T06:29:46.504ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
8142025-04-30T06:29:46.504ZDEBGtest_image_promotion (SagaRecoverer): listing candidate sagas for recovery
8152025-04-30T06:29:46.504ZTRACtest_image_promotion (SagaRecoverer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database
8162025-04-30T06:29:46.504ZDEBGtest_image_promotion (SagaRecoverer): roles roles = RoleSet { roles: {} }
8172025-04-30T06:29:46.504ZDEBGtest_image_promotion (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
8182025-04-30T06:29:46.504ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
8192025-04-30T06:29:46.504ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database
8202025-04-30T06:29:46.504ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
8212025-04-30T06:29:46.505ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
8222025-04-30T06:29:46.505ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
8232025-04-30T06:29:46.505ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
8242025-04-30T06:29:46.505ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8252025-04-30T06:29:46.506ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8262025-04-30T06:29:46.506ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
8272025-04-30T06:29:46.506ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database
8282025-04-30T06:29:46.506ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
8292025-04-30T06:29:46.507ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
8302025-04-30T06:29:46.507ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
8312025-04-30T06:29:46.507ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
8322025-04-30T06:29:46.507ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8332025-04-30T06:29:46.507ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8342025-04-30T06:29:46.507ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
8352025-04-30T06:29:46.507ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database
8362025-04-30T06:29:46.507ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
8372025-04-30T06:29:46.508ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
8382025-04-30T06:29:46.508ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
8392025-04-30T06:29:46.508ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
8402025-04-30T06:29:46.508ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8412025-04-30T06:29:46.508ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
8422025-04-30T06:29:46.508ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
8432025-04-30T06:29:46.508ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
8442025-04-30T06:29:46.509ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
8452025-04-30T06:29:46.509ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database
8462025-04-30T06:29:46.509ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
8472025-04-30T06:29:46.509ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
8482025-04-30T06:29:46.509ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 1 reason = Timeout
8492025-04-30T06:29:46.509ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
8502025-04-30T06:29:46.509ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
8512025-04-30T06:29:46.510ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
8522025-04-30T06:29:46.510ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = webhook_deliverator iteration = 1 reason = Timeout
8532025-04-30T06:29:46.510ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database
8542025-04-30T06:29:46.510ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
8552025-04-30T06:29:46.510ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
8562025-04-30T06:29:46.510ZDEBGtest_image_promotion (omicron_sled_agent::sim::Server): contacting server nexus sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8572025-04-30T06:29:46.511ZDEBGtest_image_promotion (NexusClient): client request body = Some(Body) method = POST sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = http://127.0.0.1:45630/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8582025-04-30T06:29:46.512ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
8592025-04-30T06:29:46.512ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8602025-04-30T06:29:46.512ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
8612025-04-30T06:29:46.512ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8622025-04-30T06:29:46.513ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8632025-04-30T06:29:46.514ZINFOtest_image_promotion (dropshot_internal): accepted connection local_addr = 127.0.0.1:45630 remote_addr = 127.0.0.1:55552
8642025-04-30T06:29:46.514ZTRACtest_image_promotion (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 2e 16 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:62794 req_id = b931ec06-b29a-407b-b573-39005fdbe507
8652025-04-30T06:29:46.514ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11798,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62794 req_id = b931ec06-b29a-407b-b573-39005fdbe507
8662025-04-30T06:29:46.515ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11798, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_nameservice._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 12, 95, 110, 97, 109, 101, 115, 101, 114, 118, 105, 99, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_nameservice._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62794 req_id = b931ec06-b29a-407b-b573-39005fdbe507
8672025-04-30T06:29:46.515ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
8682025-04-30T06:29:46.515ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8692025-04-30T06:29:46.515ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
8702025-04-30T06:29:46.515ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8712025-04-30T06:29:46.516ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8722025-04-30T06:29:46.516ZWARNtest_image_promotion (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_internal dns_group = internal error = no record found for Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8732025-04-30T06:29:46.516ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 94.770083ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_nameservice._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8742025-04-30T06:29:46.516ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:55552 req_id = 0d76d713-f405-402a-927e-4152234aa9b0 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8752025-04-30T06:29:46.516ZTRACtest_image_promotion (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: b4 91 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58789 req_id = 7083acde-3be4-45f5-a166-324be14379f9
8762025-04-30T06:29:46.517ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46225,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58789 req_id = 7083acde-3be4-45f5-a166-324be14379f9
8772025-04-30T06:29:46.517ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46225, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_external-dns._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 13, 95, 101, 120, 116, 101, 114, 110, 97, 108, 45, 100, 110, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_external-dns._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58789 req_id = 7083acde-3be4-45f5-a166-324be14379f9
8782025-04-30T06:29:46.519ZINFOtest_image_promotion (nexus): registered sled agent sled_uuid = b6d65341-167c-41df-9b5c-41cded99c229
8792025-04-30T06:29:46.521ZWARNtest_image_promotion (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_external dns_group = external error = no record found for Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8802025-04-30T06:29:46.521ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 33.108637ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_external-dns._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8812025-04-30T06:29:46.529ZINFOtest_image_promotion (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-04-30 06:19:46.489048467 UTC pruned = {}
8822025-04-30T06:29:46.529ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 40.226298ms iteration = 1 status = {"expiration":"2025-04-30T06:19:46.489048467Z","pruned":[]}
8832025-04-30T06:29:46.529ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8842025-04-30T06:29:46.536ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8852025-04-30T06:29:46.541ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 32 02 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 2............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:55665 req_id = a4d37f89-d1b4-4887-a88c-68b4ccb43d78
8862025-04-30T06:29:46.541ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12802,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55665 req_id = a4d37f89-d1b4-4887-a88c-68b4ccb43d78
8872025-04-30T06:29:46.541ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12802, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55665 req_id = a4d37f89-d1b4-4887-a88c-68b4ccb43d78
8882025-04-30T06:29:46.541ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: ab 96 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:49327 req_id = bd17cefd-8509-4bcc-987c-a2866b8aee36
8892025-04-30T06:29:46.541ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43926,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49327 req_id = bd17cefd-8509-4bcc-987c-a2866b8aee36
8902025-04-30T06:29:46.541ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43926, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49327 req_id = bd17cefd-8509-4bcc-987c-a2866b8aee36
8912025-04-30T06:29:46.551ZERROtest_image_promotion (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8922025-04-30T06:29:46.551ZERROtest_image_promotion (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8932025-04-30T06:29:46.552ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 61.06646ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8942025-04-30T06:29:46.552ZERROtest_image_promotion (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8952025-04-30T06:29:46.552ZERROtest_image_promotion (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8962025-04-30T06:29:46.552ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 60.728828ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8972025-04-30T06:29:46.560ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 68.789128ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
8982025-04-30T06:29:46.688ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
8992025-04-30T06:29:46.688ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9002025-04-30T06:29:46.689ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9012025-04-30T06:29:46.731ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9022025-04-30T06:29:46.735ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9032025-04-30T06:29:46.735ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9042025-04-30T06:29:46.735ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9052025-04-30T06:29:46.735ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9062025-04-30T06:29:46.795ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
9072025-04-30T06:29:46.795ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9082025-04-30T06:29:46.796ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9092025-04-30T06:29:46.801ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9102025-04-30T06:29:46.806ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9112025-04-30T06:29:46.806ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
9122025-04-30T06:29:46.806ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9132025-04-30T06:29:46.806ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9142025-04-30T06:29:46.821ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9152025-04-30T06:29:46.826ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9162025-04-30T06:29:46.826ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
9172025-04-30T06:29:46.826ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9182025-04-30T06:29:46.826ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9192025-04-30T06:29:46.827ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 322.623365ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
9202025-04-30T06:29:46.829ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9212025-04-30T06:29:46.833ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9222025-04-30T06:29:46.833ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
9232025-04-30T06:29:46.833ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9242025-04-30T06:29:46.833ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9252025-04-30T06:29:46.839ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9262025-04-30T06:29:46.842ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
9272025-04-30T06:29:46.842ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
9282025-04-30T06:29:46.842ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9292025-04-30T06:29:46.843ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
9302025-04-30T06:29:46.843ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9312025-04-30T06:29:46.846ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9322025-04-30T06:29:46.846ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
9332025-04-30T06:29:46.846ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9342025-04-30T06:29:46.847ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9352025-04-30T06:29:46.847ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9362025-04-30T06:29:46.850ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9372025-04-30T06:29:46.850ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
9382025-04-30T06:29:46.850ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9392025-04-30T06:29:46.850ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
9402025-04-30T06:29:46.850ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9412025-04-30T06:29:46.851ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9422025-04-30T06:29:46.851ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
9432025-04-30T06:29:46.851ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9442025-04-30T06:29:46.851ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9452025-04-30T06:29:46.853ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9462025-04-30T06:29:46.856ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9472025-04-30T06:29:46.856ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
9482025-04-30T06:29:46.856ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9492025-04-30T06:29:46.856ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9502025-04-30T06:29:46.868ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9512025-04-30T06:29:46.871ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9522025-04-30T06:29:46.871ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
9532025-04-30T06:29:46.871ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9542025-04-30T06:29:46.871ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
9552025-04-30T06:29:46.871ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9562025-04-30T06:29:46.871ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9572025-04-30T06:29:46.954ZDEBGtest_image_promotion (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
9582025-04-30T06:29:46.954ZINFOtest_image_promotion (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
9592025-04-30T06:29:46.954ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 450.597156ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
9602025-04-30T06:29:46.973ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 465.508385ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
9612025-04-30T06:29:46.991ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 481.444487ms iteration = 1 status = {"by_rx":{},"error":null}
9622025-04-30T06:29:46.991ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 485.397994ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
9632025-04-30T06:29:47.003ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9642025-04-30T06:29:47.027ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9652025-04-30T06:29:47.031ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9662025-04-30T06:29:47.031ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9672025-04-30T06:29:47.033ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9682025-04-30T06:29:47.048ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
9692025-04-30T06:29:47.048ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9702025-04-30T06:29:47.048ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9712025-04-30T06:29:47.048ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9722025-04-30T06:29:47.048ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9732025-04-30T06:29:47.049ZTRACtest_image_promotion (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = instance failed total_found = 0
9742025-04-30T06:29:47.049ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
9752025-04-30T06:29:47.050ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
9762025-04-30T06:29:47.050ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9772025-04-30T06:29:47.067ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
9782025-04-30T06:29:47.067ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9792025-04-30T06:29:47.068ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9802025-04-30T06:29:47.068ZWARNtest_image_promotion (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker
9812025-04-30T06:29:47.069ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 574.828288ms iteration = 1 status = {"error":"inventory collection is None"}
9822025-04-30T06:29:47.069ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
9832025-04-30T06:29:47.069ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9842025-04-30T06:29:47.069ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9852025-04-30T06:29:47.096ZWARNtest_image_promotion (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set
9862025-04-30T06:29:47.096ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 604.527711ms iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
9872025-04-30T06:29:47.097ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 588.094479ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
9882025-04-30T06:29:47.107ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9892025-04-30T06:29:47.126ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9902025-04-30T06:29:47.126ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
9912025-04-30T06:29:47.126ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9922025-04-30T06:29:47.126ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9932025-04-30T06:29:47.154ZINFOtest_image_promotion (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
9942025-04-30T06:29:47.154ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 654.288217ms iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
9952025-04-30T06:29:47.154ZINFOtest_image_promotion (SagaRecoverer): listed in-progress sagas count = 0
9962025-04-30T06:29:47.154ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 650.394318ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
9972025-04-30T06:29:47.154ZWARNtest_image_promotion (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0
9982025-04-30T06:29:47.154ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 666.717591ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
9992025-04-30T06:29:47.160ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
10002025-04-30T06:29:47.162ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
10012025-04-30T06:29:47.165ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b5 8b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61824 req_id = 46312495-a195-40e3-b8b6-a84381d06131
10022025-04-30T06:29:47.166ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46475,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61824 req_id = 46312495-a195-40e3-b8b6-a84381d06131
10032025-04-30T06:29:47.166ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46475, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61824 req_id = 46312495-a195-40e3-b8b6-a84381d06131
10042025-04-30T06:29:47.167ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
10052025-04-30T06:29:47.167ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 674.961486ms iteration = 1 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
10062025-04-30T06:29:47.168ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 668.757706ms iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10072025-04-30T06:29:47.168ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10082025-04-30T06:29:47.172ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10092025-04-30T06:29:47.172ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
10102025-04-30T06:29:47.172ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10112025-04-30T06:29:47.173ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10122025-04-30T06:29:47.174ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 679.491563ms iteration = 1 status = {}
10132025-04-30T06:29:47.183ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10142025-04-30T06:29:47.187ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10152025-04-30T06:29:47.187ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10162025-04-30T06:29:47.187ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10172025-04-30T06:29:47.187ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10182025-04-30T06:29:47.200ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
10192025-04-30T06:29:47.200ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
10202025-04-30T06:29:47.200ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10212025-04-30T06:29:47.200ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10222025-04-30T06:29:47.201ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10232025-04-30T06:29:47.227ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10242025-04-30T06:29:47.232ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
10252025-04-30T06:29:47.232ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
10262025-04-30T06:29:47.232ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10272025-04-30T06:29:47.233ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10282025-04-30T06:29:47.245ZTRACtest_image_promotion (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = webhook_dispatcher
10292025-04-30T06:29:47.245ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
10302025-04-30T06:29:47.245ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
10312025-04-30T06:29:47.246ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
10322025-04-30T06:29:47.272ZWARNtest_image_promotion (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0
10332025-04-30T06:29:47.272ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 851.581471ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
10342025-04-30T06:29:47.318ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
10352025-04-30T06:29:47.319ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10362025-04-30T06:29:47.319ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10372025-04-30T06:29:47.319ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10382025-04-30T06:29:47.320ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10392025-04-30T06:29:47.320ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Signaled
10402025-04-30T06:29:47.320ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
10412025-04-30T06:29:47.320ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10422025-04-30T06:29:47.320ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10432025-04-30T06:29:47.320ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10442025-04-30T06:29:47.321ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10452025-04-30T06:29:47.321ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 804805 local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:55552 req_id = 0d76d713-f405-402a-927e-4152234aa9b0 response_code = 204 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
10462025-04-30T06:29:47.321ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = region_replacement elapsed = 823.061526ms iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10472025-04-30T06:29:47.322ZDEBGtest_image_promotion (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:45630/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229", status: 204, headers: {"x-request-id": "0d76d713-f405-402a-927e-4152234aa9b0", "date": "Wed, 30 Apr 2025 06:29:46 GMT"} }) sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10482025-04-30T06:29:47.322ZDEBGtest_image_promotion (init_with_steps): Running step start_crucible_pantry
10492025-04-30T06:29:47.324ZDEBGtest_image_promotion (dropshot): registered endpoint kind = pantry local_addr = [::1]:65415 method = GET path = /crucible/pantry/0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10502025-04-30T06:29:47.324ZDEBGtest_image_promotion (dropshot): registered endpoint kind = pantry local_addr = [::1]:65415 method = GET path = /crucible/pantry/0/job/{id}/is-finished sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10512025-04-30T06:29:47.324ZDEBGtest_image_promotion (dropshot): registered endpoint kind = pantry local_addr = [::1]:65415 method = GET path = /crucible/pantry/0/job/{id}/ok sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10522025-04-30T06:29:47.324ZDEBGtest_image_promotion (dropshot): registered endpoint kind = pantry local_addr = [::1]:65415 method = DELETE path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10532025-04-30T06:29:47.324ZDEBGtest_image_promotion (dropshot): registered endpoint kind = pantry local_addr = [::1]:65415 method = GET path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10542025-04-30T06:29:47.324ZDEBGtest_image_promotion (dropshot): registered endpoint kind = pantry local_addr = [::1]:65415 method = POST path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10552025-04-30T06:29:47.324ZDEBGtest_image_promotion (dropshot): registered endpoint kind = pantry local_addr = [::1]:65415 method = POST path = /crucible/pantry/0/volume/{id}/background sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10562025-04-30T06:29:47.324ZDEBGtest_image_promotion (dropshot): registered endpoint kind = pantry local_addr = [::1]:65415 method = POST path = /crucible/pantry/0/volume/{id}/bulk-write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10572025-04-30T06:29:47.324ZDEBGtest_image_promotion (dropshot): registered endpoint kind = pantry local_addr = [::1]:65415 method = POST path = /crucible/pantry/0/volume/{id}/import-from-url sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10582025-04-30T06:29:47.324ZDEBGtest_image_promotion (dropshot): registered endpoint kind = pantry local_addr = [::1]:65415 method = POST path = /crucible/pantry/0/volume/{id}/scrub sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10592025-04-30T06:29:47.324ZDEBGtest_image_promotion (dropshot): registered endpoint kind = pantry local_addr = [::1]:65415 method = POST path = /crucible/pantry/0/volume/{id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10602025-04-30T06:29:47.324ZINFOtest_image_promotion (dropshot): listening kind = pantry local_addr = [::1]:65415 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10612025-04-30T06:29:47.325ZDEBGtest_image_promotion (dropshot): successfully registered DTrace USDT probes kind = pantry local_addr = [::1]:65415 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10622025-04-30T06:29:47.325ZINFOtest_image_promotion (omicron_sled_agent::sim::Server): Started Simulated Crucible Pantry address = [::1]:65415 kind = pantry sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10632025-04-30T06:29:47.326ZDEBGtest_image_promotion (init_with_steps): Running step populate_internal_dns
10642025-04-30T06:29:47.326ZDEBGtest_image_promotion: Populating Internal DNS
10652025-04-30T06:29:47.401ZINFOtest_image_promotion: DNS population: DnsConfigParams { generation: Generation( 1, ), time_created: 2025-04-30T06:29:47.401288255Z, zones: [ DnsConfigZone { zone_name: "control-plane.oxide.internal", records: { "a69550cd-17cf-4769-b40b-06fad3e3da70.host": [ Aaaa( ::1, ), ], "_cockroach._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 45371, target: "87e380f1-5fbf-414f-a2df-3a728410fb4f.host.control-plane.oxide.internal", }, ), ], "b6d65341-167c-41df-9b5c-41cded99c229.sled": [ Aaaa( ::1, ), ], "_crucible-pantry._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 65415, target: "a69550cd-17cf-4769-b40b-06fad3e3da70.host.control-plane.oxide.internal", }, ), ], "_mgd._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 40166, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "87e380f1-5fbf-414f-a2df-3a728410fb4f.host": [ Aaaa( ::1, ), ], "eee898b7-e19c-4b72-b057-42bd9105b19d.host": [ Aaaa( ::1, ), ], "_clickhouse._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 39355, target: "eee898b7-e19c-4b72-b057-42bd9105b19d.host.control-plane.oxide.internal", }, ), ], "_clickhouse-admin-single-server._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 8888, target: "eee898b7-e19c-4b72-b057-42bd9105b19d.host.control-plane.oxide.internal", }, ), ], "_nexus._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 45630, target: "d9b51ab0-9dce-44da-9764-8fd37f3b0498.host.control-plane.oxide.internal", }, ), ], "d9b51ab0-9dce-44da-9764-8fd37f3b0498.host": [ Aaaa( ::ffff:127.0.0.1, ), ], "_clickhouse-native._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "eee898b7-e19c-4b72-b057-42bd9105b19d.host.control-plane.oxide.internal", }, ), ], "bc263bb9-670b-4189-802a-5cfd0b1be255.host": [ Aaaa( ::1, ), ], "_repo-depot._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 41286, target: "b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal", }, ), ], "_external-dns._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 38208, target: "bc263bb9-670b-4189-802a-5cfd0b1be255.host.control-plane.oxide.internal", }, ), ], "_mgs._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 59233, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "bb87e4d3-5b14-4850-bbea-e9ebb9606808.host": [ Aaaa( ::1, ), ], "_oximeter-reader._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "eee898b7-e19c-4b72-b057-42bd9105b19d.host.control-plane.oxide.internal", }, ), ], "_nameservice._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 37283, target: "bb87e4d3-5b14-4850-bbea-e9ebb9606808.host.control-plane.oxide.internal", }, ), ], "_dendrite._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 41733, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host": [ Aaaa( ::1, ), ], }, }, ], }
10662025-04-30T06:29:47.402ZDEBGtest_image_promotion: client request body = Some(Body) method = PUT uri = http://[::1]:37283/config
10672025-04-30T06:29:47.406ZINFOtest_image_promotion (http): accepted connection kind = dns local_addr = [::1]:37283 remote_addr = [::1]:43171
10682025-04-30T06:29:47.407ZTRACtest_image_promotion (http): incoming request kind = dns local_addr = [::1]:37283 method = PUT remote_addr = [::1]:43171 req_id = f785a0e8-7a31-42cc-8fd4-00edf5c72018 uri = /config
10692025-04-30T06:29:47.408ZINFOtest_image_promotion (store): attempting generation update new_generation = 1 req_id = f785a0e8-7a31-42cc-8fd4-00edf5c72018
10702025-04-30T06:29:47.408ZINFOtest_image_promotion (store): pruning trees for generations newer than 0
10712025-04-30T06:29:47.408ZDEBGtest_image_promotion (store): creating tree tree_name = generation_1_zone_control-plane.oxide.internal
10722025-04-30T06:29:47.409ZDEBGtest_image_promotion (store): updating current config
10732025-04-30T06:29:47.410ZDEBGtest_image_promotion (store): flushing default tree
10742025-04-30T06:29:47.410ZINFOtest_image_promotion (store): pruning trees for generations older than 1 keep = 3
10752025-04-30T06:29:47.410ZINFOtest_image_promotion (store): updated generation new_generation = 1 req_id = f785a0e8-7a31-42cc-8fd4-00edf5c72018
10762025-04-30T06:29:47.410ZINFOtest_image_promotion (http): request completed kind = dns latency_us = 2572 local_addr = [::1]:37283 method = PUT remote_addr = [::1]:43171 req_id = f785a0e8-7a31-42cc-8fd4-00edf5c72018 response_code = 204 uri = /config
10772025-04-30T06:29:47.410ZDEBGtest_image_promotion: client response result = Ok(Response { url: "http://[::1]:37283/config", status: 204, headers: {"x-request-id": "f785a0e8-7a31-42cc-8fd4-00edf5c72018", "date": "Wed, 30 Apr 2025 06:29:46 GMT"} })
10782025-04-30T06:29:47.412ZDEBGtest_image_promotion (init_with_steps): Running step configure_sled_agents
10792025-04-30T06:29:47.481ZDEBGtest_image_promotion: client request body = Some(Body) method = PUT uri = http://[::1]:60070/omicron-config
10802025-04-30T06:29:47.488ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:61697 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10812025-04-30T06:29:47.488ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = PUT remote_addr = [::1]:61697 req_id = 49451054-7b99-443e-bfcc-72eef9e87188 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10822025-04-30T06:29:47.491ZINFOtest_image_promotion (dropshot): request completed latency_us = 3288 local_addr = [::1]:60070 method = PUT remote_addr = [::1]:61697 req_id = 49451054-7b99-443e-bfcc-72eef9e87188 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10832025-04-30T06:29:47.492ZDEBGtest_image_promotion: client response result = Ok(Response { url: "http://[::1]:60070/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "49451054-7b99-443e-bfcc-72eef9e87188", "content-length": "26", "date": "Wed, 30 Apr 2025 06:29:46 GMT"} })
10842025-04-30T06:29:47.493ZDEBGtest_image_promotion: client request body = Some(Body) method = PUT uri = http://[::1]:60070/network-bootstore-config
10852025-04-30T06:29:47.494ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = PUT remote_addr = [::1]:61697 req_id = 83d9dc43-3626-4e48-a423-e3590ebfcc5f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10862025-04-30T06:29:47.494ZINFOtest_image_promotion (dropshot): request completed latency_us = 547 local_addr = [::1]:60070 method = PUT remote_addr = [::1]:61697 req_id = 83d9dc43-3626-4e48-a423-e3590ebfcc5f response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10872025-04-30T06:29:47.494ZDEBGtest_image_promotion: client response result = Ok(Response { url: "http://[::1]:60070/network-bootstore-config", status: 204, headers: {"x-request-id": "83d9dc43-3626-4e48-a423-e3590ebfcc5f", "date": "Wed, 30 Apr 2025 06:29:46 GMT"} })
10882025-04-30T06:29:47.502ZDEBGtest_image_promotion (init_with_steps): Running step start_nexus_external
10892025-04-30T06:29:47.503ZDEBGtest_image_promotion: Starting Nexus (external API)
10902025-04-30T06:29:50.911ZTRACtest_image_promotion (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
10912025-04-30T06:29:50.911ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10922025-04-30T06:29:50.911ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
10932025-04-30T06:29:50.911ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
10942025-04-30T06:29:50.911ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10952025-04-30T06:29:50.912ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
10962025-04-30T06:29:50.912ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10972025-04-30T06:29:50.913ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10982025-04-30T06:29:50.913ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 4.408024661s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10992025-04-30T06:29:50.913ZTRACtest_image_promotion (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = start saga failed total_found = 0
11002025-04-30T06:29:50.913ZINFOtest_image_promotion (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
11012025-04-30T06:29:50.913ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 4.41283725s iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11022025-04-30T06:29:51.005ZINFOtest_image_promotion (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
11032025-04-30T06:29:51.005ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:60070/vpc-routes
11042025-04-30T06:29:51.006ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11052025-04-30T06:29:51.006ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11062025-04-30T06:29:51.006ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11072025-04-30T06:29:51.006ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11082025-04-30T06:29:51.007ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11092025-04-30T06:29:51.086ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11102025-04-30T06:29:51.087ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11112025-04-30T06:29:51.087ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11122025-04-30T06:29:51.087ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11132025-04-30T06:29:51.087ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11142025-04-30T06:29:51.144ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
11152025-04-30T06:29:51.145ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11162025-04-30T06:29:51.145ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11172025-04-30T06:29:51.146ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11182025-04-30T06:29:51.147ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:49348 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11192025-04-30T06:29:51.172ZTRACtest_image_promotion (BackgroundTasks): webhook dispatching completed successfully background_task = webhook_dispatcher events_dispatched = 0 events_without_receivers = 0
11202025-04-30T06:29:51.173ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 4.663854655s iteration = 1 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
11212025-04-30T06:29:51.183ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11222025-04-30T06:29:51.199ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
11232025-04-30T06:29:51.202ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
11242025-04-30T06:29:51.202ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
11252025-04-30T06:29:51.203ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
11262025-04-30T06:29:51.203ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11272025-04-30T06:29:51.206ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11282025-04-30T06:29:51.206ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
11292025-04-30T06:29:51.207ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11302025-04-30T06:29:51.207ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11312025-04-30T06:29:51.221ZWARNtest_image_promotion (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone
11322025-04-30T06:29:51.221ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 4.731696726s iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
11332025-04-30T06:29:51.224ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:49348 req_id = 57848815-d736-46d6-9048-6b2335c0d10a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11342025-04-30T06:29:51.224ZINFOtest_image_promotion (dropshot): request completed latency_us = 1126 local_addr = [::1]:60070 method = GET remote_addr = [::1]:49348 req_id = 57848815-d736-46d6-9048-6b2335c0d10a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11352025-04-30T06:29:51.226ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:60070/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "57848815-d736-46d6-9048-6b2335c0d10a", "content-length": "2", "date": "Wed, 30 Apr 2025 06:29:51 GMT"} })
11362025-04-30T06:29:51.226ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
11372025-04-30T06:29:51.226ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11382025-04-30T06:29:51.227ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11392025-04-30T06:29:51.232ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11402025-04-30T06:29:51.235ZDEBGtest_image_promotion (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
11412025-04-30T06:29:51.235ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11422025-04-30T06:29:51.235ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
11432025-04-30T06:29:51.235ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11442025-04-30T06:29:51.238ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11452025-04-30T06:29:51.242ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11462025-04-30T06:29:51.242ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11472025-04-30T06:29:51.242ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11482025-04-30T06:29:51.242ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11492025-04-30T06:29:51.242ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11502025-04-30T06:29:51.243ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11512025-04-30T06:29:51.256ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11522025-04-30T06:29:51.270ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11532025-04-30T06:29:51.271ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11542025-04-30T06:29:51.271ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11552025-04-30T06:29:51.271ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11562025-04-30T06:29:51.286ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11572025-04-30T06:29:51.286ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11582025-04-30T06:29:51.286ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11592025-04-30T06:29:51.286ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11602025-04-30T06:29:51.288ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 4.781347853s iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11612025-04-30T06:29:51.288ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
11622025-04-30T06:29:51.288ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
11632025-04-30T06:29:51.291ZINFOtest_image_promotion (BackgroundTasks): No bundles to collect background_task = support_bundle_collector
11642025-04-30T06:29:51.291ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 4.798559095s iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
11652025-04-30T06:29:51.298ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 93 31 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .1..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58256 req_id = d02571b7-e3b2-4e76-811b-420814fecf52
11662025-04-30T06:29:51.298ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37681,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58256 req_id = d02571b7-e3b2-4e76-811b-420814fecf52
11672025-04-30T06:29:51.299ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
11682025-04-30T06:29:51.301ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11692025-04-30T06:29:51.301ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58256 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d02571b7-e3b2-4e76-811b-420814fecf52
11702025-04-30T06:29:51.301ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3456, tv_nsec: 99490036 } })
11712025-04-30T06:29:51.301ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
11722025-04-30T06:29:51.309ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11732025-04-30T06:29:51.313ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11742025-04-30T06:29:51.314ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11752025-04-30T06:29:51.315ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11762025-04-30T06:29:51.315ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11772025-04-30T06:29:51.318ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11782025-04-30T06:29:51.318ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11792025-04-30T06:29:51.318ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11802025-04-30T06:29:51.318ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11812025-04-30T06:29:51.319ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11822025-04-30T06:29:51.320ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11832025-04-30T06:29:51.329ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
11842025-04-30T06:29:51.342ZTRACtest_image_promotion (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
11852025-04-30T06:29:51.342ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
11862025-04-30T06:29:51.342ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
11872025-04-30T06:29:51.342ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11882025-04-30T06:29:51.342ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11892025-04-30T06:29:51.343ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11902025-04-30T06:29:51.345ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11912025-04-30T06:29:51.345ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
11922025-04-30T06:29:51.346ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11932025-04-30T06:29:51.347ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
11942025-04-30T06:29:51.347ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11952025-04-30T06:29:51.347ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11962025-04-30T06:29:51.347ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11972025-04-30T06:29:51.348ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5c fe 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e \\...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56854 req_id = ddf010b6-dcff-46cf-b9f7-5fa5ddb352d3
11982025-04-30T06:29:51.348ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23806,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56854 req_id = ddf010b6-dcff-46cf-b9f7-5fa5ddb352d3
11992025-04-30T06:29:51.348ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
12002025-04-30T06:29:51.348ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56854 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ddf010b6-dcff-46cf-b9f7-5fa5ddb352d3
12012025-04-30T06:29:51.469ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
12022025-04-30T06:29:51.472ZINFOtest_image_promotion (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
12032025-04-30T06:29:51.472ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:60070/eip-gateways
12042025-04-30T06:29:51.472ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12052025-04-30T06:29:51.484ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
12062025-04-30T06:29:51.484ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12072025-04-30T06:29:51.484ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12082025-04-30T06:29:51.485ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12092025-04-30T06:29:51.636ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:60070/v2p
12102025-04-30T06:29:51.650ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12112025-04-30T06:29:51.651ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12122025-04-30T06:29:51.653ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12132025-04-30T06:29:51.654ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12142025-04-30T06:29:51.655ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[::1]:60070/artifacts-config
12152025-04-30T06:29:51.657ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12162025-04-30T06:29:51.668ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12172025-04-30T06:29:51.668ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12182025-04-30T06:29:51.668ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12192025-04-30T06:29:51.669ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12202025-04-30T06:29:51.675ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:53750 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12212025-04-30T06:29:51.675ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:36882 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12222025-04-30T06:29:51.676ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = PUT remote_addr = [::1]:49348 req_id = dae50fac-ca1d-411d-b01d-24bcf3eabe52 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12232025-04-30T06:29:51.677ZINFOtest_image_promotion (dropshot): request completed latency_us = 1139 local_addr = [::1]:60070 method = PUT remote_addr = [::1]:49348 req_id = dae50fac-ca1d-411d-b01d-24bcf3eabe52 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12242025-04-30T06:29:51.681ZINFOtest_image_promotion (InternalApi): set cockroachdb setting setting = cluster.preserve_downgrade_option value = 22.1
12252025-04-30T06:29:51.682ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12262025-04-30T06:29:51.682ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
12272025-04-30T06:29:51.685ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12282025-04-30T06:29:51.687ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:53750 req_id = a8995e99-ec14-4db3-89ff-b1c21a00024d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12292025-04-30T06:29:51.687ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = PUT remote_addr = [::1]:36882 req_id = 31e93e4b-469b-4f40-97cf-580e8f2f23c1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12302025-04-30T06:29:51.693ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 92 7d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .}..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59168 req_id = 5c041932-6712-4640-bb3a-f9be2f93852b
12312025-04-30T06:29:51.693ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37501,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59168 req_id = 5c041932-6712-4640-bb3a-f9be2f93852b
12322025-04-30T06:29:51.694ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
12332025-04-30T06:29:51.694ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37501, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59168 req_id = 5c041932-6712-4640-bb3a-f9be2f93852b
12342025-04-30T06:29:51.694ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:60070/eip-gateways", status: 204, headers: {"x-request-id": "dae50fac-ca1d-411d-b01d-24bcf3eabe52", "date": "Wed, 30 Apr 2025 06:29:51 GMT"} })
12352025-04-30T06:29:51.696ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 5.194763637s iteration = 1 status = {}
12362025-04-30T06:29:51.699ZINFOtest_image_promotion (dropshot): request completed latency_us = 12706 local_addr = [::1]:60070 method = GET remote_addr = [::1]:53750 req_id = a8995e99-ec14-4db3-89ff-b1c21a00024d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12372025-04-30T06:29:51.700ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12382025-04-30T06:29:51.711ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12392025-04-30T06:29:51.712ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12402025-04-30T06:29:51.715ZINFOtest_image_promotion (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmp1ODgKk/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12412025-04-30T06:29:51.715ZDEBGtest_image_promotion (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmp1ODgKk/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12422025-04-30T06:29:51.715ZINFOtest_image_promotion (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmp1371pQ/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12432025-04-30T06:29:51.715ZDEBGtest_image_promotion (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmp1371pQ/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12442025-04-30T06:29:51.715ZINFOtest_image_promotion (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmp1ODgKk/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12452025-04-30T06:29:51.719ZINFOtest_image_promotion (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmp1371pQ/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12462025-04-30T06:29:51.720ZINFOtest_image_promotion (ArtifactStore): Starting delete reconciler generation = 1 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12472025-04-30T06:29:51.721ZINFOtest_image_promotion (dropshot): request completed latency_us = 33662 local_addr = [::1]:60070 method = PUT remote_addr = [::1]:36882 req_id = 31e93e4b-469b-4f40-97cf-580e8f2f23c1 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12482025-04-30T06:29:51.724ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
12492025-04-30T06:29:51.724ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
12502025-04-30T06:29:51.724ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
12512025-04-30T06:29:51.725ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:60070/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8995e99-ec14-4db3-89ff-b1c21a00024d", "content-length": "2", "date": "Wed, 30 Apr 2025 06:29:51 GMT"} })
12522025-04-30T06:29:51.725ZINFOtest_image_promotion (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
12532025-04-30T06:29:51.725ZINFOtest_image_promotion (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
12542025-04-30T06:29:51.725ZINFOtest_image_promotion (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
12552025-04-30T06:29:51.727ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 5.231746623s iteration = 1 status = {}
12562025-04-30T06:29:51.727ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:60070/artifacts-config", status: 204, headers: {"x-request-id": "31e93e4b-469b-4f40-97cf-580e8f2f23c1", "date": "Wed, 30 Apr 2025 06:29:51 GMT"} })
12572025-04-30T06:29:51.727ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = None method = GET uri = http://[::1]:60070/artifacts
12582025-04-30T06:29:51.727ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12592025-04-30T06:29:51.727ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12602025-04-30T06:29:51.728ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12612025-04-30T06:29:51.729ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
12622025-04-30T06:29:51.729ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12632025-04-30T06:29:51.729ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12642025-04-30T06:29:51.729ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
12652025-04-30T06:29:51.732ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12662025-04-30T06:29:51.754ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:36882 req_id = 5d1c6deb-203e-4a58-a864-723445737644 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12672025-04-30T06:29:51.764ZINFOtest_image_promotion (dropshot): request completed latency_us = 9906 local_addr = [::1]:60070 method = GET remote_addr = [::1]:36882 req_id = 5d1c6deb-203e-4a58-a864-723445737644 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12682025-04-30T06:29:51.767ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:60070/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d1c6deb-203e-4a58-a864-723445737644", "content-length": "26", "date": "Wed, 30 Apr 2025 06:29:51 GMT"} })
12692025-04-30T06:29:51.767ZINFOtest_image_promotion (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication sled = http://[::1]:60070
12702025-04-30T06:29:51.770ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 5.262058738s iteration = 1 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:29:51.727456162Z"},{"operation":"list","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:29:51.767468905Z"}]}
12712025-04-30T06:29:51.770ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12722025-04-30T06:29:51.770ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12732025-04-30T06:29:51.784ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12742025-04-30T06:29:51.818ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12752025-04-30T06:29:51.823ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
12762025-04-30T06:29:51.823ZDEBGtest_image_promotion (InternalApi): updating rack subnet for rack c19a698f-c6f9-4a17-ae30-20d711b8f7dc to Some( V6( Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, ), )
12772025-04-30T06:29:51.823ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12782025-04-30T06:29:51.823ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
12792025-04-30T06:29:51.824ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12802025-04-30T06:29:51.825ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12812025-04-30T06:29:51.826ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12822025-04-30T06:29:51.826ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12832025-04-30T06:29:51.826ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12842025-04-30T06:29:51.826ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12852025-04-30T06:29:51.852ZINFOtest_image_promotion (InternalApi): Using static configuration for external switchports
12862025-04-30T06:29:51.852ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12872025-04-30T06:29:51.852ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
12882025-04-30T06:29:51.853ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12892025-04-30T06:29:51.854ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c084ef05-a575-4086-895f-02f6ff94edc5", "content-length": "835", "date": "Wed, 30 Apr 2025 06:29:51 GMT"} })
12902025-04-30T06:29:51.855ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
12912025-04-30T06:29:51.857ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12922025-04-30T06:29:51.857ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12932025-04-30T06:29:51.857ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12942025-04-30T06:29:51.857ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12952025-04-30T06:29:51.866ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12962025-04-30T06:29:51.866ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12972025-04-30T06:29:51.866ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12982025-04-30T06:29:51.867ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12992025-04-30T06:29:51.870ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13002025-04-30T06:29:51.918ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
13012025-04-30T06:29:51.918ZTRACtest_image_promotion (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
13022025-04-30T06:29:51.919ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13032025-04-30T06:29:51.919ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13042025-04-30T06:29:51.920ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13052025-04-30T06:29:51.949ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13062025-04-30T06:29:51.949ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13072025-04-30T06:29:51.950ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13082025-04-30T06:29:51.976ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13092025-04-30T06:29:51.977ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13102025-04-30T06:29:51.979ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13112025-04-30T06:29:52.030ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3011095-61de-4a11-af4d-7fa6d54c506d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:51 GMT"} })
13122025-04-30T06:29:52.038ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
13132025-04-30T06:29:52.038ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13142025-04-30T06:29:52.038ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13152025-04-30T06:29:52.039ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13162025-04-30T06:29:52.040ZINFOtest_image_promotion (InternalApi): Recording Rack Network Configuration
13172025-04-30T06:29:52.040ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13182025-04-30T06:29:52.040ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
13192025-04-30T06:29:52.041ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13202025-04-30T06:29:52.062ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13212025-04-30T06:29:52.062ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13222025-04-30T06:29:52.063ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13232025-04-30T06:29:52.063ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13242025-04-30T06:29:52.064ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13252025-04-30T06:29:52.092ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13262025-04-30T06:29:52.092ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13272025-04-30T06:29:52.092ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13282025-04-30T06:29:52.092ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13292025-04-30T06:29:52.096ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13302025-04-30T06:29:52.096ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13312025-04-30T06:29:52.096ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13322025-04-30T06:29:52.097ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13332025-04-30T06:29:52.107ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13342025-04-30T06:29:52.115ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13352025-04-30T06:29:52.115ZTRACtest_image_promotion (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
13362025-04-30T06:29:52.115ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13372025-04-30T06:29:52.115ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13382025-04-30T06:29:52.115ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13392025-04-30T06:29:52.115ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13402025-04-30T06:29:52.116ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13412025-04-30T06:29:52.167ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13422025-04-30T06:29:52.167ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13432025-04-30T06:29:52.167ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13442025-04-30T06:29:52.168ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13452025-04-30T06:29:52.183ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13462025-04-30T06:29:52.186ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13472025-04-30T06:29:52.186ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13482025-04-30T06:29:52.187ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13492025-04-30T06:29:52.193ZTRACtest_image_promotion (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
13502025-04-30T06:29:52.194ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13512025-04-30T06:29:52.196ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13522025-04-30T06:29:52.198ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
13532025-04-30T06:29:52.198ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13542025-04-30T06:29:52.258ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13552025-04-30T06:29:52.270ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13562025-04-30T06:29:52.270ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13572025-04-30T06:29:52.270ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13582025-04-30T06:29:52.270ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13592025-04-30T06:29:52.323ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13602025-04-30T06:29:52.329ZDEBGtest_image_promotion (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
13612025-04-30T06:29:52.329ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13622025-04-30T06:29:52.329ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
13632025-04-30T06:29:52.330ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13642025-04-30T06:29:52.330ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f29cc265-bf2e-47c6-b838-70fe45daf1b3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
13652025-04-30T06:29:52.334ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
13662025-04-30T06:29:52.357ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13672025-04-30T06:29:52.357ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13682025-04-30T06:29:52.358ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13692025-04-30T06:29:52.366ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 785aa91b-ec3e-49d8-8a9e-6b6c1461a445, lookup_type: ByName("oxide-service-pool") }
13702025-04-30T06:29:52.366ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13712025-04-30T06:29:52.366ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13722025-04-30T06:29:52.366ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
13732025-04-30T06:29:52.366ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13742025-04-30T06:29:52.373ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13752025-04-30T06:29:52.373ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13762025-04-30T06:29:52.374ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13772025-04-30T06:29:52.399ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13782025-04-30T06:29:52.399ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13792025-04-30T06:29:52.400ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13802025-04-30T06:29:52.406ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13812025-04-30T06:29:52.406ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13822025-04-30T06:29:52.407ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13832025-04-30T06:29:52.407ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13842025-04-30T06:29:52.412ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 785aa91b-ec3e-49d8-8a9e-6b6c1461a445, lookup_type: ByName("oxide-service-pool") } result = Ok(())
13852025-04-30T06:29:52.412ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13862025-04-30T06:29:52.412ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
13872025-04-30T06:29:52.412ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13882025-04-30T06:29:52.415ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
13892025-04-30T06:29:52.415ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13902025-04-30T06:29:52.415ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13912025-04-30T06:29:52.415ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13922025-04-30T06:29:52.416ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13932025-04-30T06:29:52.419ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13942025-04-30T06:29:52.419ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13952025-04-30T06:29:52.419ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13962025-04-30T06:29:52.419ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13972025-04-30T06:29:52.421ZTRACtest_image_promotion (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 785aa91b-ec3e-49d8-8a9e-6b6c1461a445, lookup_type: ByName("oxide-service-pool") }
13982025-04-30T06:29:52.421ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13992025-04-30T06:29:52.421ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14002025-04-30T06:29:52.421ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
14012025-04-30T06:29:52.421ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14022025-04-30T06:29:52.423ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14032025-04-30T06:29:52.423ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14042025-04-30T06:29:52.423ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14052025-04-30T06:29:52.423ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14062025-04-30T06:29:52.429ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14072025-04-30T06:29:52.437ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14082025-04-30T06:29:52.437ZTRACtest_image_promotion (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
14092025-04-30T06:29:52.437ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
14102025-04-30T06:29:52.437ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
14112025-04-30T06:29:52.437ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14122025-04-30T06:29:52.437ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14132025-04-30T06:29:52.437ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14142025-04-30T06:29:52.489ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14152025-04-30T06:29:52.492ZDEBGtest_image_promotion (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 785aa91b-ec3e-49d8-8a9e-6b6c1461a445, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14162025-04-30T06:29:52.494ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14172025-04-30T06:29:52.494ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14182025-04-30T06:29:52.494ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14192025-04-30T06:29:52.495ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14202025-04-30T06:29:52.505ZTRACtest_image_promotion (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 785aa91b-ec3e-49d8-8a9e-6b6c1461a445, lookup_type: ByName("oxide-service-pool") }
14212025-04-30T06:29:52.517ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14222025-04-30T06:29:52.517ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14232025-04-30T06:29:52.517ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
14242025-04-30T06:29:52.518ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14252025-04-30T06:29:52.520ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14262025-04-30T06:29:52.520ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14272025-04-30T06:29:52.520ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14282025-04-30T06:29:52.521ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14292025-04-30T06:29:52.532ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14302025-04-30T06:29:52.535ZDEBGtest_image_promotion (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 785aa91b-ec3e-49d8-8a9e-6b6c1461a445, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14312025-04-30T06:29:52.536ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d61624b5-b997-4283-bf41-dee7b4a7e393", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14322025-04-30T06:29:52.537ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
14332025-04-30T06:29:52.540ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14342025-04-30T06:29:52.558ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14352025-04-30T06:29:52.558ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14362025-04-30T06:29:52.558ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14372025-04-30T06:29:52.558ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14382025-04-30T06:29:52.559ZTRACtest_image_promotion (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
14392025-04-30T06:29:52.559ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14402025-04-30T06:29:52.559ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14412025-04-30T06:29:52.560ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
14422025-04-30T06:29:52.560ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14432025-04-30T06:29:52.597ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14442025-04-30T06:29:52.622ZDEBGtest_image_promotion (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
14452025-04-30T06:29:52.623ZDEBGtest_image_promotion (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
14462025-04-30T06:29:52.623ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet
14472025-04-30T06:29:52.623ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14482025-04-30T06:29:52.623ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14492025-04-30T06:29:52.623ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14502025-04-30T06:29:52.624ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14512025-04-30T06:29:52.627ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14522025-04-30T06:29:52.631ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
14532025-04-30T06:29:52.631ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14542025-04-30T06:29:52.631ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14552025-04-30T06:29:52.631ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14562025-04-30T06:29:52.673ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7471d40b-b9ec-4a70-ad08-d7b9e5184607", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14572025-04-30T06:29:52.674ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
14582025-04-30T06:29:52.674ZTRACtest_image_promotion (service-firewall-rule-progator): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:06:08.248500Z, allowed_ips: None } background_task = service_firewall_rule_propagation
14592025-04-30T06:29:52.674ZDEBGtest_image_promotion (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
14602025-04-30T06:29:52.674ZDEBGtest_image_promotion (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
14612025-04-30T06:29:52.674ZDEBGtest_image_promotion (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
14622025-04-30T06:29:52.753ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e618d094-6b4e-4aa6-bc78-534c3c7c244e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14632025-04-30T06:29:52.753ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
14642025-04-30T06:29:52.756ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ef5e174-8a62-43aa-bba5-8a6feac6b8c2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14652025-04-30T06:29:52.756ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
14662025-04-30T06:29:52.757ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13472ef7-283f-4acd-9ec7-f71cdc6ff47e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14672025-04-30T06:29:52.757ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
14682025-04-30T06:29:52.759ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87bb7eac-575f-4860-aa8a-03737eda0ecd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14692025-04-30T06:29:52.766ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
14702025-04-30T06:29:52.790ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6aa0d944-696a-43c9-b18f-b3e6cc085aa4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14712025-04-30T06:29:52.808ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
14722025-04-30T06:29:52.838ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45b8520a-1976-49e1-b66b-dea28fc0b7cf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14732025-04-30T06:29:52.838ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
14742025-04-30T06:29:52.845ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1716bfd0-04f5-4b1e-b23e-39e433088781", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14752025-04-30T06:29:52.846ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
14762025-04-30T06:29:52.851ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f97a9e26-dbca-49ce-8f3f-8880c067128b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14772025-04-30T06:29:52.851ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
14782025-04-30T06:29:52.856ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcd533dc-ab80-4d25-86dd-c630654d50a3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14792025-04-30T06:29:52.856ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
14802025-04-30T06:29:52.857ZINFOtest_image_promotion (InternalApi): inserted blueprint blueprint_id = e00f4a6c-0b84-4aa1-90aa-7860edb99088
14812025-04-30T06:29:52.858ZTRACtest_image_promotion (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
14822025-04-30T06:29:52.858ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14832025-04-30T06:29:52.858ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14842025-04-30T06:29:52.858ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
14852025-04-30T06:29:52.858ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14862025-04-30T06:29:52.861ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "297e0359-3c5c-48af-988f-811827f99e4f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14872025-04-30T06:29:52.861ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
14882025-04-30T06:29:52.861ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14892025-04-30T06:29:52.866ZDEBGtest_image_promotion (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
14902025-04-30T06:29:52.870ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "edcb834b-a5af-40fa-b147-89cfacfdedfe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
14912025-04-30T06:29:52.870ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
14922025-04-30T06:29:52.871ZINFOtest_image_promotion (InternalApi): No networking records needed for cockroach_db service
14932025-04-30T06:29:52.871ZINFOtest_image_promotion (InternalApi): No networking records needed for crucible_pantry service
14942025-04-30T06:29:52.871ZINFOtest_image_promotion (InternalApi): No networking records needed for internal_dns service
14952025-04-30T06:29:52.873ZDEBGtest_image_promotion (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
14962025-04-30T06:29:52.873ZDEBGtest_image_promotion (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
14972025-04-30T06:29:52.873ZINFOtest_image_promotion (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation
14982025-04-30T06:29:52.873ZDEBGtest_image_promotion (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 6.371748622s
14992025-04-30T06:29:52.873ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 6.371810937s iteration = 1 status = {}
15002025-04-30T06:29:52.876ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddff9e30-c265-49a5-9fb5-43bbbd6c7f21", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15012025-04-30T06:29:52.876ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
15022025-04-30T06:29:52.909ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90858840-a3a7-4f81-914b-f53227cbc8a0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15032025-04-30T06:29:52.909ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
15042025-04-30T06:29:52.916ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9af6fcf7-2c32-44c0-8c43-2669fb349349", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15052025-04-30T06:29:52.917ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
15062025-04-30T06:29:52.918ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33701411-6b35-4ddc-a8a7-a05d87611470", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15072025-04-30T06:29:52.918ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
15082025-04-30T06:29:52.919ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8639626-5351-441a-9521-8b80a08285b3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15092025-04-30T06:29:52.920ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
15102025-04-30T06:29:52.928ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4caec07-2fb9-4d31-8269-248a9619d23a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15112025-04-30T06:29:52.930ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
15122025-04-30T06:29:52.963ZINFOtest_image_promotion (InternalApi): Inserted networking records for external_dns service
15132025-04-30T06:29:52.968ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b647f38-00d6-4dc0-aaad-c2762807140f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15142025-04-30T06:29:52.968ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
15152025-04-30T06:29:52.997ZINFOtest_image_promotion (InternalApi): Inserted networking records for nexus service
15162025-04-30T06:29:52.997ZINFOtest_image_promotion (InternalApi): No networking records needed for clickhouse service
15172025-04-30T06:29:52.997ZINFOtest_image_promotion (InternalApi): Inserted service networking records
15182025-04-30T06:29:52.997ZINFOtest_image_promotion (InternalApi): Inserted physical disks
15192025-04-30T06:29:52.997ZINFOtest_image_promotion (InternalApi): Inserted zpools
15202025-04-30T06:29:52.997ZINFOtest_image_promotion (InternalApi): Inserted datasets
15212025-04-30T06:29:53.005ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcaf3c2d-bd06-4a89-aa6f-bf8bc878c21e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15222025-04-30T06:29:53.005ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
15232025-04-30T06:29:53.018ZINFOtest_image_promotion (InternalApi): Populated DNS tables for internal DNS
15242025-04-30T06:29:53.024ZINFOtest_image_promotion (InternalApi): Populated DNS tables for external DNS
15252025-04-30T06:29:53.024ZTRACtest_image_promotion (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
15262025-04-30T06:29:53.025ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15272025-04-30T06:29:53.025ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15282025-04-30T06:29:53.025ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
15292025-04-30T06:29:53.026ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15302025-04-30T06:29:53.036ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9f02e95-a6f4-4854-8fc2-35d0b9681ffa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15312025-04-30T06:29:53.037ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
15322025-04-30T06:29:53.041ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "465af0cb-e06c-4e79-92f2-bcf60c6a889a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15332025-04-30T06:29:53.041ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
15342025-04-30T06:29:53.047ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15352025-04-30T06:29:53.051ZDEBGtest_image_promotion (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
15362025-04-30T06:29:53.051ZTRACtest_image_promotion (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
15372025-04-30T06:29:53.051ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15382025-04-30T06:29:53.051ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15392025-04-30T06:29:53.051ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
15402025-04-30T06:29:53.053ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15412025-04-30T06:29:53.056ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "340b6e35-e052-4990-bfc4-fe2b09a5afba", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15422025-04-30T06:29:53.056ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
15432025-04-30T06:29:53.056ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15442025-04-30T06:29:53.058ZDEBGtest_image_promotion (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
15452025-04-30T06:29:53.061ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fd65776-b5d1-44aa-9892-2af080853452", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15462025-04-30T06:29:53.061ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
15472025-04-30T06:29:53.064ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9191611d-e806-4490-a7b6-22b365e4c971", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15482025-04-30T06:29:53.064ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
15492025-04-30T06:29:53.067ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6aa67722-eb03-4cde-9a52-52f7c43a9e50", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15502025-04-30T06:29:53.067ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
15512025-04-30T06:29:53.068ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "deafd7e4-e672-447c-92bb-63db3b7d2f33", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15522025-04-30T06:29:53.068ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
15532025-04-30T06:29:53.074ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b5f9992-086f-4ed9-a066-16c905f6c036", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15542025-04-30T06:29:53.074ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
15552025-04-30T06:29:53.075ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4caea50f-74a2-491c-b56a-33ee1d4bd387", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15562025-04-30T06:29:53.075ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
15572025-04-30T06:29:53.077ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb3f8e50-cdb7-4af4-bc0c-109180e77dfe", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15582025-04-30T06:29:53.077ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
15592025-04-30T06:29:53.077ZTRACtest_image_promotion (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
15602025-04-30T06:29:53.077ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15612025-04-30T06:29:53.077ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15622025-04-30T06:29:53.077ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
15632025-04-30T06:29:53.078ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15642025-04-30T06:29:53.082ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15652025-04-30T06:29:53.102ZDEBGtest_image_promotion (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
15662025-04-30T06:29:53.102ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
15672025-04-30T06:29:53.102ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15682025-04-30T06:29:53.102ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15692025-04-30T06:29:53.102ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
15702025-04-30T06:29:53.103ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15712025-04-30T06:29:53.103ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28e6c239-1b05-43b7-a1c0-711e200996e9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15722025-04-30T06:29:53.104ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
15732025-04-30T06:29:53.107ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bc30812-b3e7-4ea7-89cc-50b1b65c1fef", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15742025-04-30T06:29:53.107ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
15752025-04-30T06:29:53.109ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfe99cbf-f2c5-4bcd-8b64-22478ed7df06", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15762025-04-30T06:29:53.109ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
15772025-04-30T06:29:53.111ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a26a2dc3-ee6e-4d96-aa89-774b6c9daf5f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15782025-04-30T06:29:53.111ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
15792025-04-30T06:29:53.113ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "931919c0-badd-46e0-9953-5204796cfb67", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15802025-04-30T06:29:53.113ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
15812025-04-30T06:29:53.115ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "246ad879-f675-4235-b441-09342feab734", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15822025-04-30T06:29:53.115ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
15832025-04-30T06:29:53.117ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52fc71d8-bcfd-4434-a3c8-6e6ee963c13c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15842025-04-30T06:29:53.117ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
15852025-04-30T06:29:53.118ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a60be412-6153-431d-b5c3-e8fee24eac47", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15862025-04-30T06:29:53.118ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
15872025-04-30T06:29:53.119ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aafc7731-427b-4997-8fca-241454352ba3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15882025-04-30T06:29:53.119ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
15892025-04-30T06:29:53.120ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45a9f9dc-f9eb-4b32-ad1b-fe743e3aa63e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15902025-04-30T06:29:53.120ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
15912025-04-30T06:29:53.121ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "672eb2ff-cc80-404b-b283-a63295488285", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:52 GMT"} })
15922025-04-30T06:29:53.121ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15932025-04-30T06:29:53.121ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15942025-04-30T06:29:53.121ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15952025-04-30T06:29:53.121ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15962025-04-30T06:29:53.122ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15972025-04-30T06:29:53.158ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15982025-04-30T06:29:53.183ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
15992025-04-30T06:29:53.188ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16002025-04-30T06:29:53.196ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
16012025-04-30T06:29:53.203ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
16022025-04-30T06:29:53.203ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16032025-04-30T06:29:53.203ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
16042025-04-30T06:29:53.203ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
16052025-04-30T06:29:53.204ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
16062025-04-30T06:29:53.208ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16072025-04-30T06:29:53.208ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16082025-04-30T06:29:53.208ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16092025-04-30T06:29:53.208ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16102025-04-30T06:29:53.209ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16112025-04-30T06:29:53.214ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16122025-04-30T06:29:53.222ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
16132025-04-30T06:29:53.222ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
16142025-04-30T06:29:53.222ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
16152025-04-30T06:29:53.222ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
16162025-04-30T06:29:53.226ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16172025-04-30T06:29:53.233ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16182025-04-30T06:29:53.284ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
16192025-04-30T06:29:53.284ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
16202025-04-30T06:29:53.285ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:49839 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
16212025-04-30T06:29:53.285ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:49839 req_id = d4a32c87-0128-4265-9264-0fb57a050caa sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
16222025-04-30T06:29:53.286ZINFOtest_image_promotion (dropshot): request completed latency_us = 521 local_addr = [::1]:60070 method = GET remote_addr = [::1]:49839 req_id = d4a32c87-0128-4265-9264-0fb57a050caa response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
16232025-04-30T06:29:53.287ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4a32c87-0128-4265-9264-0fb57a050caa", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:29:53 GMT"} })
16242025-04-30T06:29:53.289ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
16252025-04-30T06:29:53.289ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
16262025-04-30T06:29:53.289ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
16272025-04-30T06:29:53.291ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
16282025-04-30T06:29:53.291ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16292025-04-30T06:29:53.291ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
16302025-04-30T06:29:53.291ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
16312025-04-30T06:29:53.294ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
16322025-04-30T06:29:53.298ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16332025-04-30T06:29:53.303ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
16342025-04-30T06:29:53.303ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
16352025-04-30T06:29:53.303ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
16362025-04-30T06:29:53.304ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
16372025-04-30T06:29:53.310ZINFOtest_image_promotion (InternalApi): Created recovery silo
16382025-04-30T06:29:53.313ZINFOtest_image_promotion (InternalApi): Created recovery user
16392025-04-30T06:29:53.324ZINFOtest_image_promotion (InternalApi): Created recovery user's password
16402025-04-30T06:29:53.324ZTRACtest_image_promotion (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: f93c0e80-61a0-4aa9-a766-5bb7291705c8, lookup_type: ById(f93c0e80-61a0-4aa9-a766-5bb7291705c8) }
16412025-04-30T06:29:53.324ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = f93c0e80-61a0-4aa9-a766-5bb7291705c8 resource_type = Silo
16422025-04-30T06:29:53.324ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16432025-04-30T06:29:53.324ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16442025-04-30T06:29:53.325ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16452025-04-30T06:29:53.329ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16462025-04-30T06:29:53.329ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16472025-04-30T06:29:53.329ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16482025-04-30T06:29:53.330ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16492025-04-30T06:29:53.332ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16502025-04-30T06:29:53.335ZDEBGtest_image_promotion (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: f93c0e80-61a0-4aa9-a766-5bb7291705c8, lookup_type: ById(f93c0e80-61a0-4aa9-a766-5bb7291705c8) } result = Ok(())
16512025-04-30T06:29:53.335ZDEBGtest_image_promotion (InternalApi): Generated role assignment queries
16522025-04-30T06:29:53.364ZINFOtest_image_promotion (InternalApi): Granted Silo privileges
16532025-04-30T06:29:53.364ZTRACtest_image_promotion (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16542025-04-30T06:29:53.364ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16552025-04-30T06:29:53.364ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16562025-04-30T06:29:53.364ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16572025-04-30T06:29:53.365ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16582025-04-30T06:29:53.377ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16592025-04-30T06:29:53.380ZDEBGtest_image_promotion (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16602025-04-30T06:29:53.398ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16612025-04-30T06:29:53.398ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16622025-04-30T06:29:53.400ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16632025-04-30T06:29:53.403ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16642025-04-30T06:29:53.403ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16652025-04-30T06:29:53.404ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16662025-04-30T06:29:53.411ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16672025-04-30T06:29:53.411ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16682025-04-30T06:29:53.412ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16692025-04-30T06:29:53.420ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16702025-04-30T06:29:53.420ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16712025-04-30T06:29:53.420ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16722025-04-30T06:29:53.420ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16732025-04-30T06:29:53.420ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16742025-04-30T06:29:53.431ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16752025-04-30T06:29:53.431ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16762025-04-30T06:29:53.431ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16772025-04-30T06:29:53.433ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16782025-04-30T06:29:53.437ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16792025-04-30T06:29:53.437ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16802025-04-30T06:29:53.437ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16812025-04-30T06:29:53.437ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16822025-04-30T06:29:53.441ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16832025-04-30T06:29:53.456ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
16842025-04-30T06:29:53.456ZTRACtest_image_promotion (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
16852025-04-30T06:29:53.456ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16862025-04-30T06:29:53.456ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16872025-04-30T06:29:53.456ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16882025-04-30T06:29:53.456ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16892025-04-30T06:29:53.457ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16902025-04-30T06:29:53.460ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16912025-04-30T06:29:53.460ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16922025-04-30T06:29:53.460ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16932025-04-30T06:29:53.461ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16942025-04-30T06:29:53.463ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16952025-04-30T06:29:53.464ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16962025-04-30T06:29:53.464ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
16972025-04-30T06:29:53.464ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16982025-04-30T06:29:53.466ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16992025-04-30T06:29:53.473ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17002025-04-30T06:29:53.473ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17012025-04-30T06:29:53.473ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17022025-04-30T06:29:53.473ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17032025-04-30T06:29:53.477ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17042025-04-30T06:29:53.478ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17052025-04-30T06:29:53.478ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17062025-04-30T06:29:53.483ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17072025-04-30T06:29:53.483ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17082025-04-30T06:29:53.484ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17092025-04-30T06:29:53.492ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17102025-04-30T06:29:53.492ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17112025-04-30T06:29:53.493ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17122025-04-30T06:29:53.539ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
17132025-04-30T06:29:53.539ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17142025-04-30T06:29:53.540ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17152025-04-30T06:29:53.541ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17162025-04-30T06:29:53.544ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17172025-04-30T06:29:53.556ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17182025-04-30T06:29:53.556ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17192025-04-30T06:29:53.556ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17202025-04-30T06:29:53.556ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17212025-04-30T06:29:53.560ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17222025-04-30T06:29:53.560ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17232025-04-30T06:29:53.560ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17242025-04-30T06:29:53.560ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17252025-04-30T06:29:53.564ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17262025-04-30T06:29:53.576ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17272025-04-30T06:29:53.576ZTRACtest_image_promotion (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
17282025-04-30T06:29:53.576ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17292025-04-30T06:29:53.576ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17302025-04-30T06:29:53.576ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17312025-04-30T06:29:53.580ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17322025-04-30T06:29:53.581ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17332025-04-30T06:29:53.581ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17342025-04-30T06:29:53.583ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17352025-04-30T06:29:53.584ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17362025-04-30T06:29:53.584ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17372025-04-30T06:29:53.587ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17382025-04-30T06:29:53.587ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17392025-04-30T06:29:53.588ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17402025-04-30T06:29:53.596ZTRACtest_image_promotion (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17412025-04-30T06:29:53.596ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17422025-04-30T06:29:53.596ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17432025-04-30T06:29:53.596ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17442025-04-30T06:29:53.597ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17452025-04-30T06:29:53.599ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17462025-04-30T06:29:53.599ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17472025-04-30T06:29:53.599ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17482025-04-30T06:29:53.600ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17492025-04-30T06:29:53.602ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17502025-04-30T06:29:53.602ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17512025-04-30T06:29:53.602ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17522025-04-30T06:29:53.602ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17532025-04-30T06:29:53.604ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17542025-04-30T06:29:53.611ZDEBGtest_image_promotion (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17552025-04-30T06:29:53.611ZTRACtest_image_promotion (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
17562025-04-30T06:29:53.611ZTRACtest_image_promotion (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17572025-04-30T06:29:53.611ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17582025-04-30T06:29:53.611ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17592025-04-30T06:29:53.611ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17602025-04-30T06:29:53.611ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17612025-04-30T06:29:53.613ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17622025-04-30T06:29:53.614ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17632025-04-30T06:29:53.614ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17642025-04-30T06:29:53.615ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17652025-04-30T06:29:53.617ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17662025-04-30T06:29:53.617ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17672025-04-30T06:29:53.617ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17682025-04-30T06:29:53.618ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17692025-04-30T06:29:53.620ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17702025-04-30T06:29:53.626ZDEBGtest_image_promotion (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17712025-04-30T06:29:53.626ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17722025-04-30T06:29:53.626ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17732025-04-30T06:29:53.627ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17742025-04-30T06:29:53.631ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17752025-04-30T06:29:53.631ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17762025-04-30T06:29:53.632ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17772025-04-30T06:29:53.634ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17782025-04-30T06:29:53.634ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17792025-04-30T06:29:53.634ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17802025-04-30T06:29:53.636ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17812025-04-30T06:29:53.636ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17822025-04-30T06:29:53.637ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17832025-04-30T06:29:53.640ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17842025-04-30T06:29:53.640ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17852025-04-30T06:29:53.640ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17862025-04-30T06:29:53.642ZTRACtest_image_promotion (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
17872025-04-30T06:29:53.642ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17882025-04-30T06:29:53.642ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17892025-04-30T06:29:53.642ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17902025-04-30T06:29:53.643ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17912025-04-30T06:29:53.644ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17922025-04-30T06:29:53.645ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17932025-04-30T06:29:53.645ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17942025-04-30T06:29:53.645ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17952025-04-30T06:29:53.647ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17962025-04-30T06:29:53.647ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17972025-04-30T06:29:53.647ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
17982025-04-30T06:29:53.648ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17992025-04-30T06:29:53.655ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18002025-04-30T06:29:53.666ZDEBGtest_image_promotion (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18012025-04-30T06:29:53.666ZTRACtest_image_promotion (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
18022025-04-30T06:29:53.666ZTRACtest_image_promotion (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
18032025-04-30T06:29:53.666ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
18042025-04-30T06:29:53.666ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18052025-04-30T06:29:53.666ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
18062025-04-30T06:29:53.667ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18072025-04-30T06:29:53.669ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18082025-04-30T06:29:53.669ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18092025-04-30T06:29:53.669ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
18102025-04-30T06:29:53.669ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18112025-04-30T06:29:53.671ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18122025-04-30T06:29:53.671ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18132025-04-30T06:29:53.672ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
18142025-04-30T06:29:53.672ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18152025-04-30T06:29:53.675ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18162025-04-30T06:29:53.694ZDEBGtest_image_promotion (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18172025-04-30T06:29:53.694ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18182025-04-30T06:29:53.694ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
18192025-04-30T06:29:53.694ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18202025-04-30T06:29:53.710ZDEBGtest_image_promotion (nexus): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: bf77b0f7-839e-436d-8370-ef5ca8a0ebae, kind: Service { id: d9b51ab0-9dce-44da-9764-8fd37f3b0498 }, name: Name("nexus-d9b51ab0-9dce-44da-9764-8fd37f3b0498"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 3d9a262e-f7ce-40cb-9f2f-7421ff0f6ec6, kind: Service { id: bc263bb9-670b-4189-802a-5cfd0b1be255 }, name: Name("external-dns-bc263bb9-670b-4189-802a-5cfd0b1be255"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
18212025-04-30T06:29:53.710ZTRACtest_image_promotion (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
18222025-04-30T06:29:53.710ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18232025-04-30T06:29:53.710ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18242025-04-30T06:29:53.710ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
18252025-04-30T06:29:53.714ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18262025-04-30T06:29:53.716ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18272025-04-30T06:29:53.720ZDEBGtest_image_promotion (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
18282025-04-30T06:29:53.720ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18292025-04-30T06:29:53.720ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
18302025-04-30T06:29:53.720ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18312025-04-30T06:29:53.720ZINFOtest_image_promotion (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = a1711333-5886-4224-9910-e4f59cbe995f
18322025-04-30T06:29:53.720ZDEBGtest_image_promotion (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = a1711333-5886-4224-9910-e4f59cbe995f time_started = 2025-04-30 06:29:51.724193 UTC
18332025-04-30T06:29:53.720ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 6.400121343s iteration = 2 status = {"collection_id":"a1711333-5886-4224-9910-e4f59cbe995f","time_done":"2025-04-30 06:29:53.289034 UTC","time_started":"2025-04-30 06:29:51.724193 UTC"}
18342025-04-30T06:29:53.721ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
18352025-04-30T06:29:53.721ZWARNtest_image_promotion (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
18362025-04-30T06:29:53.721ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 619.805\u{b5}s iteration = 2 status = {"error":"no blueprint"}
18372025-04-30T06:29:53.721ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
18382025-04-30T06:29:53.721ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 13.397\u{b5}s iteration = 2 status = {"error":"task disabled"}
18392025-04-30T06:29:53.723ZTRACtest_image_promotion (nexus): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:29:53.380695Z, allowed_ips: None }
18402025-04-30T06:29:53.723ZDEBGtest_image_promotion (nexus): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
18412025-04-30T06:29:53.724ZDEBGtest_image_promotion (nexus): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 3d9a262e-f7ce-40cb-9f2f-7421ff0f6ec6, kind: Service { id: bc263bb9-670b-4189-802a-5cfd0b1be255 }, name: Name("external-dns-bc263bb9-670b-4189-802a-5cfd0b1be255"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: bf77b0f7-839e-436d-8370-ef5ca8a0ebae, kind: Service { id: d9b51ab0-9dce-44da-9764-8fd37f3b0498 }, name: Name("nexus-d9b51ab0-9dce-44da-9764-8fd37f3b0498"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
18422025-04-30T06:29:53.724ZDEBGtest_image_promotion (nexus): resolved 2 rules for sleds
18432025-04-30T06:29:53.746ZDEBGtest_image_promotion (nexus): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:29:46.519511Z, time_modified: 2025-04-30T06:29:46.519511Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(60070), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(41286) }]
18442025-04-30T06:29:53.746ZDEBGtest_image_promotion (nexus): sending firewall rules to sled agents
18452025-04-30T06:29:53.746ZTRACtest_image_promotion (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18462025-04-30T06:29:53.746ZDEBGtest_image_promotion (InstanceAllocator): roles roles = RoleSet { roles: {} }
18472025-04-30T06:29:53.747ZDEBGtest_image_promotion (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18482025-04-30T06:29:53.750ZTRACtest_image_promotion (InstanceAllocator): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
18492025-04-30T06:29:53.750ZTRACtest_image_promotion (InstanceAllocator): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18502025-04-30T06:29:53.750ZTRACtest_image_promotion (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18512025-04-30T06:29:53.750ZDEBGtest_image_promotion (InstanceAllocator): roles roles = RoleSet { roles: {} }
18522025-04-30T06:29:53.751ZDEBGtest_image_promotion (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18532025-04-30T06:29:53.753ZDEBGtest_image_promotion (InstanceAllocator): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "viewer")} }
18542025-04-30T06:29:53.756ZDEBGtest_image_promotion (InstanceAllocator): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
18552025-04-30T06:29:53.812ZDEBGtest_image_promotion (nexus): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:60070/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18562025-04-30T06:29:53.813ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:51968 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
18572025-04-30T06:29:53.814ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = PUT remote_addr = [::1]:51968 req_id = 41b6e5c9-9120-4307-b09a-634a6b2fb2b7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18582025-04-30T06:29:53.821ZINFOtest_image_promotion (dropshot): request completed latency_us = 7409 local_addr = [::1]:60070 method = PUT remote_addr = [::1]:51968 req_id = 41b6e5c9-9120-4307-b09a-634a6b2fb2b7 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18592025-04-30T06:29:53.822ZDEBGtest_image_promotion (nexus): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:60070/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "41b6e5c9-9120-4307-b09a-634a6b2fb2b7", "date": "Wed, 30 Apr 2025 06:29:53 GMT"} })
18602025-04-30T06:29:53.824ZINFOtest_image_promotion (nexus): updated firewall rules on 1 sleds
18612025-04-30T06:29:53.825ZTRACtest_image_promotion (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
18622025-04-30T06:29:53.825ZDEBGtest_image_promotion (ServiceBalancer): roles roles = RoleSet { roles: {} }
18632025-04-30T06:29:53.826ZDEBGtest_image_promotion (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
18642025-04-30T06:29:53.826ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Signaled
18652025-04-30T06:29:53.826ZDEBGtest_image_promotion (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18662025-04-30T06:29:53.826ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
18672025-04-30T06:29:53.826ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18682025-04-30T06:29:53.827ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18692025-04-30T06:29:53.827ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Signaled
18702025-04-30T06:29:53.828ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
18712025-04-30T06:29:53.828ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Signaled
18722025-04-30T06:29:53.828ZDEBGtest_image_promotion (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18732025-04-30T06:29:53.828ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
18742025-04-30T06:29:53.828ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18752025-04-30T06:29:53.829ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18762025-04-30T06:29:53.829ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Signaled
18772025-04-30T06:29:53.829ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
18782025-04-30T06:29:53.829ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Signaled
18792025-04-30T06:29:53.829ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
18802025-04-30T06:29:53.829ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18812025-04-30T06:29:53.829ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
18822025-04-30T06:29:53.829ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18832025-04-30T06:29:53.830ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18842025-04-30T06:29:53.830ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 3 reason = Signaled
18852025-04-30T06:29:53.830ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
18862025-04-30T06:29:53.830ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18872025-04-30T06:29:53.830ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
18882025-04-30T06:29:53.830ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
18892025-04-30T06:29:53.831ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
18902025-04-30T06:29:53.831ZTRACtest_image_promotion (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 59 ba 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 Y............_na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:55195 req_id = e64d6d91-2bf5-43d0-b1da-60bc9a7817d2
18912025-04-30T06:29:53.831ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22970,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55195 req_id = e64d6d91-2bf5-43d0-b1da-60bc9a7817d2
18922025-04-30T06:29:53.832ZDEBGtest_image_promotion (store): query key key = _nameservice._tcp
18932025-04-30T06:29:53.832ZDEBGtest_image_promotion (store): query key key = bb87e4d3-5b14-4850-bbea-e9ebb9606808.host
18942025-04-30T06:29:53.832ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("bb87e4d3-5b14-4850-bbea-e9ebb9606808.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55195 query = LowerQuery { name: LowerName(Name("_nameservice._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 37283, target: Name("bb87e4d3-5b14-4850-bbea-e9ebb9606808.host.control-plane.oxide.internal") })) }] req_id = e64d6d91-2bf5-43d0-b1da-60bc9a7817d2
18952025-04-30T06:29:53.832ZTRACtest_image_promotion (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 72 66 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 rf..........._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63293 req_id = 5dde8ed3-8f8e-4dc0-9a90-4993f757ad85
18962025-04-30T06:29:53.832ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29286,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63293 req_id = 5dde8ed3-8f8e-4dc0-9a90-4993f757ad85
18972025-04-30T06:29:53.833ZDEBGtest_image_promotion (store): query key key = _external-dns._tcp
18982025-04-30T06:29:53.833ZDEBGtest_image_promotion (store): query key key = bc263bb9-670b-4189-802a-5cfd0b1be255.host
18992025-04-30T06:29:53.833ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("bc263bb9-670b-4189-802a-5cfd0b1be255.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63293 query = LowerQuery { name: LowerName(Name("_external-dns._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 38208, target: Name("bc263bb9-670b-4189-802a-5cfd0b1be255.host.control-plane.oxide.internal") })) }] req_id = 5dde8ed3-8f8e-4dc0-9a90-4993f757ad85
19002025-04-30T06:29:53.835ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 37283, target: Name("bb87e4d3-5b14-4850-bbea-e9ebb9606808.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bb87e4d3-5b14-4850-bbea-e9ebb9606808.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3458, tv_nsec: 634656839 } })
19012025-04-30T06:29:53.835ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
19022025-04-30T06:29:53.837ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 38208, target: Name("bc263bb9-670b-4189-802a-5cfd0b1be255.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bc263bb9-670b-4189-802a-5cfd0b1be255.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3458, tv_nsec: 635902180 } })
19032025-04-30T06:29:53.837ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
19042025-04-30T06:29:53.837ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19052025-04-30T06:29:53.837ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19062025-04-30T06:29:53.837ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19072025-04-30T06:29:53.837ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19082025-04-30T06:29:53.837ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19092025-04-30T06:29:53.838ZTRACtest_image_promotion (ServiceBalancer): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
19102025-04-30T06:29:53.838ZTRACtest_image_promotion (ServiceBalancer): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19112025-04-30T06:29:53.838ZTRACtest_image_promotion (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
19122025-04-30T06:29:53.838ZDEBGtest_image_promotion (ServiceBalancer): roles roles = RoleSet { roles: {} }
19132025-04-30T06:29:53.838ZDEBGtest_image_promotion (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
19142025-04-30T06:29:53.838ZTRACtest_image_promotion (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: ee 8f 01 00 00 01 00 00 00 00 00 01 24 62 62 38 ............$bb8\n0010: 37 65 34 64 33 2d 35 62 31 34 2d 34 38 35 30 2d 7e4d3-5b14-4850-\n0020: 62 62 65 61 2d 65 39 65 62 62 39 36 30 36 38 30 bbea-e9ebb960680\n0030: 38 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 8.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:58821 req_id = 344d7926-dab2-403e-a0f0-fdb38ae8dae1
19152025-04-30T06:29:53.838ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61071,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("bb87e4d3-5b14-4850-bbea-e9ebb9606808.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("bb87e4d3-5b14-4850-bbea-e9ebb9606808.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 98,\n 98,\n 56,\n 55,\n 101,\n 52,\n 100,\n 51,\n 45,\n 53,\n 98,\n 49,\n 52,\n 45,\n 52,\n 56,\n 53,\n 48,\n 45,\n 98,\n 98,\n 101,\n 97,\n 45,\n 101,\n 57,\n 101,\n 98,\n 98,\n 57,\n 54,\n 48,\n 54,\n 56,\n 48,\n 56,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58821 req_id = 344d7926-dab2-403e-a0f0-fdb38ae8dae1
19162025-04-30T06:29:53.839ZDEBGtest_image_promotion (store): query key key = bb87e4d3-5b14-4850-bbea-e9ebb9606808.host
19172025-04-30T06:29:53.839ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58821 query = LowerQuery { name: LowerName(Name("bb87e4d3-5b14-4850-bbea-e9ebb9606808.host.control-plane.oxide.internal.")), original: Query { name: Name("bb87e4d3-5b14-4850-bbea-e9ebb9606808.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("bb87e4d3-5b14-4850-bbea-e9ebb9606808.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 344d7926-dab2-403e-a0f0-fdb38ae8dae1
19182025-04-30T06:29:53.839ZTRACtest_image_promotion (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: b8 74 01 00 00 01 00 00 00 00 00 01 24 62 63 32 .t..........$bc2\n0010: 36 33 62 62 39 2d 36 37 30 62 2d 34 31 38 39 2d 63bb9-670b-4189-\n0020: 38 30 32 61 2d 35 63 66 64 30 62 31 62 65 32 35 802a-5cfd0b1be25\n0030: 35 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 5.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:52023 req_id = 7655cd16-dd73-4267-97b5-7d3cf16a322a
19192025-04-30T06:29:53.839ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47220,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("bc263bb9-670b-4189-802a-5cfd0b1be255.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("bc263bb9-670b-4189-802a-5cfd0b1be255.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 98,\n 99,\n 50,\n 54,\n 51,\n 98,\n 98,\n 57,\n 45,\n 54,\n 55,\n 48,\n 98,\n 45,\n 52,\n 49,\n 56,\n 57,\n 45,\n 56,\n 48,\n 50,\n 97,\n 45,\n 53,\n 99,\n 102,\n 100,\n 48,\n 98,\n 49,\n 98,\n 101,\n 50,\n 53,\n 53,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52023 req_id = 7655cd16-dd73-4267-97b5-7d3cf16a322a
19202025-04-30T06:29:53.839ZDEBGtest_image_promotion (store): query key key = bc263bb9-670b-4189-802a-5cfd0b1be255.host
19212025-04-30T06:29:53.839ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52023 query = LowerQuery { name: LowerName(Name("bc263bb9-670b-4189-802a-5cfd0b1be255.host.control-plane.oxide.internal.")), original: Query { name: Name("bc263bb9-670b-4189-802a-5cfd0b1be255.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("bc263bb9-670b-4189-802a-5cfd0b1be255.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7655cd16-dd73-4267-97b5-7d3cf16a322a
19222025-04-30T06:29:53.839ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19232025-04-30T06:29:53.839ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19242025-04-30T06:29:53.839ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19252025-04-30T06:29:53.839ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19262025-04-30T06:29:53.840ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19272025-04-30T06:29:53.841ZINFOtest_image_promotion (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:37283] } background_task = dns_servers_internal dns_group = internal
19282025-04-30T06:29:53.841ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 13.182766ms iteration = 2 status = {"addresses":["[::1]:37283"]}
19292025-04-30T06:29:53.841ZINFOtest_image_promotion (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:38208] } background_task = dns_servers_external dns_group = external
19302025-04-30T06:29:53.841ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 11.595584ms iteration = 2 status = {"addresses":["[::1]:38208"]}
19312025-04-30T06:29:53.841ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
19322025-04-30T06:29:53.841ZWARNtest_image_promotion (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config
19332025-04-30T06:29:53.841ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 21.383\u{b5}s iteration = 2 status = {"error":"no config"}
19342025-04-30T06:29:53.841ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
19352025-04-30T06:29:53.841ZWARNtest_image_promotion (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config
19362025-04-30T06:29:53.841ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 9.288\u{b5}s iteration = 2 status = {"error":"no config"}
19372025-04-30T06:29:53.845ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19382025-04-30T06:29:53.850ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19392025-04-30T06:29:53.850ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19402025-04-30T06:29:53.850ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19412025-04-30T06:29:53.851ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19422025-04-30T06:29:53.851ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19432025-04-30T06:29:53.854ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
19442025-04-30T06:29:53.854ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19452025-04-30T06:29:53.854ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19462025-04-30T06:29:53.855ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19472025-04-30T06:29:53.857ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19482025-04-30T06:29:53.863ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19492025-04-30T06:29:53.864ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19502025-04-30T06:29:53.870ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19512025-04-30T06:29:53.871ZDEBGtest_image_promotion (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19522025-04-30T06:29:53.871ZDEBGtest_image_promotion (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19532025-04-30T06:29:53.871ZDEBGtest_image_promotion (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
19542025-04-30T06:29:53.871ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19552025-04-30T06:29:53.871ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19562025-04-30T06:29:53.871ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19572025-04-30T06:29:53.871ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19582025-04-30T06:29:53.872ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19592025-04-30T06:29:53.872ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
19602025-04-30T06:29:53.872ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19612025-04-30T06:29:53.872ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19622025-04-30T06:29:53.872ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19632025-04-30T06:29:53.872ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19642025-04-30T06:29:53.874ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
19652025-04-30T06:29:53.874ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
19662025-04-30T06:29:53.875ZDEBGtest_image_promotion (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19672025-04-30T06:29:53.875ZDEBGtest_image_promotion (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19682025-04-30T06:29:53.875ZDEBGtest_image_promotion (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19692025-04-30T06:29:53.875ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19702025-04-30T06:29:53.875ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19712025-04-30T06:29:53.875ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19722025-04-30T06:29:53.875ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19732025-04-30T06:29:53.875ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19742025-04-30T06:29:53.875ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19752025-04-30T06:29:53.881ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19762025-04-30T06:29:53.882ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19772025-04-30T06:29:53.882ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19782025-04-30T06:29:53.882ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19792025-04-30T06:29:53.885ZDEBGtest_image_promotion (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19802025-04-30T06:29:53.893ZDEBGtest_image_promotion (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
19812025-04-30T06:29:53.893ZINFOtest_image_promotion (nexus): Rack initialized
19822025-04-30T06:29:53.893ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19832025-04-30T06:29:53.893ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
19842025-04-30T06:29:53.894ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19852025-04-30T06:29:53.896ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19862025-04-30T06:29:53.903ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19872025-04-30T06:29:53.903ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19882025-04-30T06:29:53.903ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19892025-04-30T06:29:53.903ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19902025-04-30T06:29:53.905ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19912025-04-30T06:29:53.914ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19922025-04-30T06:29:53.914ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19932025-04-30T06:29:53.914ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19942025-04-30T06:29:53.914ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19952025-04-30T06:29:53.915ZDEBGtest_image_promotion (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
19962025-04-30T06:29:53.915ZDEBGtest_image_promotion (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 13b3627d-2cdf-40be-a4ad-9be137f66f91 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
19972025-04-30T06:29:53.915ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19982025-04-30T06:29:53.915ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19992025-04-30T06:29:53.915ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20002025-04-30T06:29:53.915ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20012025-04-30T06:29:53.916ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20022025-04-30T06:29:53.917ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ac 96 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51243 req_id = d47228f6-efa1-4ee3-b4b4-bb0fce2dc7ba
20032025-04-30T06:29:53.917ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44182,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51243 req_id = d47228f6-efa1-4ee3-b4b4-bb0fce2dc7ba
20042025-04-30T06:29:53.918ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
20052025-04-30T06:29:53.918ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
20062025-04-30T06:29:53.918ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51243 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d47228f6-efa1-4ee3-b4b4-bb0fce2dc7ba
20072025-04-30T06:29:53.919ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3458, tv_nsec: 718285985 } })
20082025-04-30T06:29:53.919ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
20092025-04-30T06:29:53.921ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b1 d1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58220 req_id = 0e94a286-71f6-4347-80f4-3305579f2743
20102025-04-30T06:29:53.921ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45521,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58220 req_id = 0e94a286-71f6-4347-80f4-3305579f2743
20112025-04-30T06:29:53.921ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
20122025-04-30T06:29:53.921ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58220 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0e94a286-71f6-4347-80f4-3305579f2743
20132025-04-30T06:29:53.997ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
20142025-04-30T06:29:53.999ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20152025-04-30T06:29:53.999ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
20162025-04-30T06:29:54.003ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20172025-04-30T06:29:54.003ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
20182025-04-30T06:29:54.003ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20192025-04-30T06:29:54.003ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20202025-04-30T06:29:54.003ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20212025-04-30T06:29:54.004ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20222025-04-30T06:29:54.005ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a9 8b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64386 req_id = 3bc1c000-a99d-43ad-8647-b74683fca596
20232025-04-30T06:29:54.006ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43403,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64386 req_id = 3bc1c000-a99d-43ad-8647-b74683fca596
20242025-04-30T06:29:54.006ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
20252025-04-30T06:29:54.006ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43403, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64386 req_id = 3bc1c000-a99d-43ad-8647-b74683fca596
20262025-04-30T06:29:54.008ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
20272025-04-30T06:29:54.008ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
20282025-04-30T06:29:54.008ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
20292025-04-30T06:29:54.008ZDEBGtest_image_promotion (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20302025-04-30T06:29:54.008ZDEBGtest_image_promotion (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 2d42a667-9691-45aa-a1db-59239aca916b dns_zone_name = oxide-dev.test found_so_far = 0 version = 2
20312025-04-30T06:29:54.008ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
20322025-04-30T06:29:54.009ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20332025-04-30T06:29:54.009ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20342025-04-30T06:29:54.009ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20352025-04-30T06:29:54.009ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20362025-04-30T06:29:54.011ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20372025-04-30T06:29:54.021ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20382025-04-30T06:29:54.022ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20392025-04-30T06:29:54.022ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20402025-04-30T06:29:54.023ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20412025-04-30T06:29:54.023ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20422025-04-30T06:29:54.023ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
20432025-04-30T06:29:54.024ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20442025-04-30T06:29:54.040ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20452025-04-30T06:29:54.045ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20462025-04-30T06:29:54.045ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20472025-04-30T06:29:54.045ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20482025-04-30T06:29:54.045ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20492025-04-30T06:29:54.046ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20502025-04-30T06:29:54.052ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20512025-04-30T06:29:54.053ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20522025-04-30T06:29:54.053ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20532025-04-30T06:29:54.054ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20542025-04-30T06:29:54.056ZDEBGtest_image_promotion (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 13b3627d-2cdf-40be-a4ad-9be137f66f91 dns_zone_name = control-plane.oxide.internal found_so_far = 21 version = 1
20552025-04-30T06:29:54.056ZDEBGtest_image_promotion (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20562025-04-30T06:29:54.056ZINFOtest_image_promotion (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal generation = 1
20572025-04-30T06:29:54.056ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 229.949139ms iteration = 2 status = {"generation":1}
20582025-04-30T06:29:54.056ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Dependency
20592025-04-30T06:29:54.107ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[::1]:37283 generation = 1 method = PUT servers = DnsServersList { addresses: [[::1]:37283] } uri = http://[::1]:37283/config
20602025-04-30T06:29:54.109ZINFOtest_image_promotion (http): accepted connection kind = dns local_addr = [::1]:37283 remote_addr = [::1]:33817
20612025-04-30T06:29:54.113ZTRACtest_image_promotion (http): incoming request kind = dns local_addr = [::1]:37283 method = PUT remote_addr = [::1]:33817 req_id = 1ed8e9db-06c2-4f9c-858b-c35793f7b3fe uri = /config
20622025-04-30T06:29:54.116ZINFOtest_image_promotion (store): attempting generation update new_generation = 1 req_id = 1ed8e9db-06c2-4f9c-858b-c35793f7b3fe
20632025-04-30T06:29:54.117ZINFOtest_image_promotion (store): updated generation new_generation = 1 req_id = 1ed8e9db-06c2-4f9c-858b-c35793f7b3fe
20642025-04-30T06:29:54.117ZINFOtest_image_promotion (http): request completed kind = dns latency_us = 3385 local_addr = [::1]:37283 method = PUT remote_addr = [::1]:33817 req_id = 1ed8e9db-06c2-4f9c-858b-c35793f7b3fe response_code = 204 uri = /config
20652025-04-30T06:29:54.119ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:37283 generation = 1 result = Ok(Response { url: "http://[::1]:37283/config", status: 204, headers: {"x-request-id": "1ed8e9db-06c2-4f9c-858b-c35793f7b3fe", "date": "Wed, 30 Apr 2025 06:29:53 GMT"} }) servers = DnsServersList { addresses: [[::1]:37283] }
20662025-04-30T06:29:54.119ZINFOtest_image_promotion (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:37283 generation = 1 servers = DnsServersList { addresses: [[::1]:37283] }
20672025-04-30T06:29:54.121ZINFOtest_image_promotion (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal generation = 1 servers = DnsServersList { addresses: [[::1]:37283] }
20682025-04-30T06:29:54.121ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 64.838917ms iteration = 3 status = {"generation":1,"server_results":{"[::1]:37283":{"Ok":null}}}
20692025-04-30T06:29:54.122ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
20702025-04-30T06:29:54.122ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
20712025-04-30T06:29:54.122ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20722025-04-30T06:29:54.122ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
20732025-04-30T06:29:54.123ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20742025-04-30T06:29:54.147ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f77872f-8eb6-487f-9185-fc60104eae2a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:29:53 GMT"} })
20752025-04-30T06:29:54.147ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
20762025-04-30T06:29:54.148ZDEBGtest_image_promotion (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 2d42a667-9691-45aa-a1db-59239aca916b dns_zone_name = oxide-dev.test found_so_far = 1 version = 2
20772025-04-30T06:29:54.148ZDEBGtest_image_promotion (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20782025-04-30T06:29:54.148ZINFOtest_image_promotion (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external generation = 2
20792025-04-30T06:29:54.148ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 319.333351ms iteration = 2 status = {"generation":2}
20802025-04-30T06:29:54.148ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Dependency
20812025-04-30T06:29:54.199ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[::1]:38208 generation = 2 method = PUT servers = DnsServersList { addresses: [[::1]:38208] } uri = http://[::1]:38208/config
20822025-04-30T06:29:54.202ZINFOtest_image_promotion (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-04-30T06:29:45.054114Z, time_modified: 2025-04-30T06:29:45.054114Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "test-suite-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: f93c0e80-61a0-4aa9-a766-5bb7291705c8, db_silo: Silo { identity: SiloIdentity { id: f93c0e80-61a0-4aa9-a766-5bb7291705c8, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:29:53.024831Z, time_modified: 2025-04-30T06:29:53.024831Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide-dev.test" }, NoSiloCerts { silo_id: f93c0e80-61a0-4aa9-a766-5bb7291705c8, dns_name: "test-suite-silo.sys.oxide-dev.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: f93c0e80-61a0-4aa9-a766-5bb7291705c8, db_silo: Silo { identity: SiloIdentity { id: f93c0e80-61a0-4aa9-a766-5bb7291705c8, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:29:53.024831Z, time_modified: 2025-04-30T06:29:53.024831Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }) }
20832025-04-30T06:29:54.202ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 372.485864ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide-dev.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"test-suite-silo.sys.oxide-dev.test":{"silo_id":"f93c0e80-61a0-4aa9-a766-5bb7291705c8","tls_certs":[]}},"default_endpoint":{"silo_id":"f93c0e80-61a0-4aa9-a766-5bb7291705c8","tls_certs":[]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide-dev.test\\" has no usable certificates","silo f93c0e80-61a0-4aa9-a766-5bb7291705c8 with DNS name \\"test-suite-silo.sys.oxide-dev.test\\" has no usable certificates"]}
20842025-04-30T06:29:54.204ZINFOtest_image_promotion (http): accepted connection kind = dns local_addr = [::1]:38208 remote_addr = [::1]:42722
20852025-04-30T06:29:54.214ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
20862025-04-30T06:29:54.214ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20872025-04-30T06:29:54.214ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
20882025-04-30T06:29:54.215ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20892025-04-30T06:29:54.215ZTRACtest_image_promotion (http): incoming request kind = dns local_addr = [::1]:38208 method = PUT remote_addr = [::1]:42722 req_id = 87f62f11-ddd9-4936-b8df-7bb08f6a357b uri = /config
20902025-04-30T06:29:54.216ZINFOtest_image_promotion (store): attempting generation update new_generation = 2 req_id = 87f62f11-ddd9-4936-b8df-7bb08f6a357b
20912025-04-30T06:29:54.216ZINFOtest_image_promotion (store): pruning trees for generations newer than 0
20922025-04-30T06:29:54.216ZDEBGtest_image_promotion (store): creating tree tree_name = generation_2_zone_oxide-dev.test
20932025-04-30T06:29:54.216ZDEBGtest_image_promotion (store): updating current config
20942025-04-30T06:29:54.216ZDEBGtest_image_promotion (store): flushing default tree
20952025-04-30T06:29:54.216ZINFOtest_image_promotion (store): pruning trees for generations older than 2 keep = 3
20962025-04-30T06:29:54.216ZINFOtest_image_promotion (store): updated generation new_generation = 2 req_id = 87f62f11-ddd9-4936-b8df-7bb08f6a357b
20972025-04-30T06:29:54.217ZINFOtest_image_promotion (http): request completed kind = dns latency_us = 1636 local_addr = [::1]:38208 method = PUT remote_addr = [::1]:42722 req_id = 87f62f11-ddd9-4936-b8df-7bb08f6a357b response_code = 204 uri = /config
20982025-04-30T06:29:54.222ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:38208 generation = 2 result = Ok(Response { url: "http://[::1]:38208/config", status: 204, headers: {"x-request-id": "87f62f11-ddd9-4936-b8df-7bb08f6a357b", "date": "Wed, 30 Apr 2025 06:29:53 GMT"} }) servers = DnsServersList { addresses: [[::1]:38208] }
20992025-04-30T06:29:54.222ZINFOtest_image_promotion (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:38208 generation = 2 servers = DnsServersList { addresses: [[::1]:38208] }
21002025-04-30T06:29:54.222ZINFOtest_image_promotion (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external generation = 2 servers = DnsServersList { addresses: [[::1]:38208] }
21012025-04-30T06:29:54.222ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 73.820042ms iteration = 3 status = {"generation":2,"server_results":{"[::1]:38208":{"Ok":null}}}
21022025-04-30T06:29:54.223ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21032025-04-30T06:29:54.223ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21042025-04-30T06:29:54.223ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21052025-04-30T06:29:54.224ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21062025-04-30T06:29:54.228ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a300586-d21e-4290-8c0d-851fdbed9ed7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:53 GMT"} })
21072025-04-30T06:29:54.229ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
21082025-04-30T06:29:54.229ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21092025-04-30T06:29:54.237ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21102025-04-30T06:29:54.237ZTRACtest_image_promotion (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21112025-04-30T06:29:54.237ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21122025-04-30T06:29:54.238ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21132025-04-30T06:29:54.238ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21142025-04-30T06:29:54.238ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21152025-04-30T06:29:54.239ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21162025-04-30T06:29:54.249ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21172025-04-30T06:29:54.249ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21182025-04-30T06:29:54.249ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21192025-04-30T06:29:54.249ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21202025-04-30T06:29:54.252ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21212025-04-30T06:29:54.252ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21222025-04-30T06:29:54.252ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21232025-04-30T06:29:54.253ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21242025-04-30T06:29:54.255ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21252025-04-30T06:29:54.263ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21262025-04-30T06:29:54.263ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21272025-04-30T06:29:54.263ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21282025-04-30T06:29:54.263ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21292025-04-30T06:29:54.264ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e98f6224-504a-484c-a7dc-675f8d93263b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:53 GMT"} })
21302025-04-30T06:29:54.264ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
21312025-04-30T06:29:54.268ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21322025-04-30T06:29:54.268ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21332025-04-30T06:29:54.269ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21342025-04-30T06:29:54.272ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27f39fa2-b809-46bc-8aba-e5dd73ae5632", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:53 GMT"} })
21352025-04-30T06:29:54.273ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
21362025-04-30T06:29:54.274ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21372025-04-30T06:29:54.274ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21382025-04-30T06:29:54.275ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21392025-04-30T06:29:54.277ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21402025-04-30T06:29:54.277ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21412025-04-30T06:29:54.279ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21422025-04-30T06:29:54.281ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fdb9a3d-93ad-4c57-bd81-ece220fa4b89", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:53 GMT"} })
21432025-04-30T06:29:54.281ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
21442025-04-30T06:29:54.282ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21452025-04-30T06:29:54.282ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21462025-04-30T06:29:54.282ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21472025-04-30T06:29:54.282ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21482025-04-30T06:29:54.284ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21492025-04-30T06:29:54.286ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21502025-04-30T06:29:54.286ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21512025-04-30T06:29:54.286ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21522025-04-30T06:29:54.287ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21532025-04-30T06:29:54.287ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db5b275e-b9f1-4f3f-963c-91a23bb0cc00", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
21542025-04-30T06:29:54.287ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
21552025-04-30T06:29:54.289ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21562025-04-30T06:29:54.289ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21572025-04-30T06:29:54.289ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21582025-04-30T06:29:54.290ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21592025-04-30T06:29:54.291ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0561f7b6-a7b8-4b60-a2c4-dee7d689b9bd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
21602025-04-30T06:29:54.291ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
21612025-04-30T06:29:54.292ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21622025-04-30T06:29:54.299ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21632025-04-30T06:29:54.299ZTRACtest_image_promotion (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21642025-04-30T06:29:54.299ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21652025-04-30T06:29:54.299ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21662025-04-30T06:29:54.300ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21672025-04-30T06:29:54.303ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21682025-04-30T06:29:54.303ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21692025-04-30T06:29:54.304ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21702025-04-30T06:29:54.304ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50a6249c-4675-4554-beb2-07af83213bba", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
21712025-04-30T06:29:54.305ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
21722025-04-30T06:29:54.308ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21732025-04-30T06:29:54.308ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21742025-04-30T06:29:54.308ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21752025-04-30T06:29:54.309ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61ff772e-1702-4133-ae6d-5ba3019d300d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
21762025-04-30T06:29:54.309ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
21772025-04-30T06:29:54.311ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fee6f56-8253-4afa-8add-757884290bff", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
21782025-04-30T06:29:54.311ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
21792025-04-30T06:29:54.312ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21802025-04-30T06:29:54.312ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21812025-04-30T06:29:54.312ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21822025-04-30T06:29:54.315ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b90d1437-96bc-4457-9e25-978894187bb4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
21832025-04-30T06:29:54.315ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
21842025-04-30T06:29:54.316ZTRACtest_image_promotion (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
21852025-04-30T06:29:54.316ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21862025-04-30T06:29:54.316ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21872025-04-30T06:29:54.317ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21882025-04-30T06:29:54.317ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21892025-04-30T06:29:54.320ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de48ee99-b25f-434d-a837-967aff7212bd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
21902025-04-30T06:29:54.320ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
21912025-04-30T06:29:54.321ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21922025-04-30T06:29:54.321ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21932025-04-30T06:29:54.321ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
21942025-04-30T06:29:54.322ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21952025-04-30T06:29:54.325ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43bb3773-fd28-4a37-ad79-007de0e62cf9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
21962025-04-30T06:29:54.325ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
21972025-04-30T06:29:54.326ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21982025-04-30T06:29:54.326ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21992025-04-30T06:29:54.326ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22002025-04-30T06:29:54.326ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22012025-04-30T06:29:54.329ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22022025-04-30T06:29:54.346ZDEBGtest_image_promotion (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22032025-04-30T06:29:54.347ZTRACtest_image_promotion (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22042025-04-30T06:29:54.347ZTRACtest_image_promotion (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22052025-04-30T06:29:54.347ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22062025-04-30T06:29:54.347ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22072025-04-30T06:29:54.347ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22082025-04-30T06:29:54.347ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22092025-04-30T06:29:54.356ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87d2ea34-8887-4667-a7b8-7ea62ef716c8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22102025-04-30T06:29:54.356ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
22112025-04-30T06:29:54.357ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22122025-04-30T06:29:54.357ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22132025-04-30T06:29:54.358ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22142025-04-30T06:29:54.358ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22152025-04-30T06:29:54.360ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e48e497-29d0-4d80-b338-b9b233babcb5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22162025-04-30T06:29:54.360ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
22172025-04-30T06:29:54.361ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22182025-04-30T06:29:54.361ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22192025-04-30T06:29:54.361ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22202025-04-30T06:29:54.363ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22212025-04-30T06:29:54.365ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d416e98-b32d-4f8b-9908-a3411f74e4a3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22222025-04-30T06:29:54.365ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
22232025-04-30T06:29:54.366ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22242025-04-30T06:29:54.375ZDEBGtest_image_promotion (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22252025-04-30T06:29:54.376ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22262025-04-30T06:29:54.376ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22272025-04-30T06:29:54.376ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22282025-04-30T06:29:54.381ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22292025-04-30T06:29:54.381ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22302025-04-30T06:29:54.382ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22312025-04-30T06:29:54.384ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52a2f10c-266d-4489-9b1b-f7568f3664a9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22322025-04-30T06:29:54.385ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
22332025-04-30T06:29:54.385ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22342025-04-30T06:29:54.385ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22352025-04-30T06:29:54.386ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22362025-04-30T06:29:54.389ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5150d387-9627-4e44-828b-ec648a55daef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22372025-04-30T06:29:54.389ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
22382025-04-30T06:29:54.389ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22392025-04-30T06:29:54.389ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22402025-04-30T06:29:54.390ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22412025-04-30T06:29:54.393ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91cd6cf1-78ed-4412-984e-1b20bfc3290d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22422025-04-30T06:29:54.393ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
22432025-04-30T06:29:54.393ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22442025-04-30T06:29:54.393ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22452025-04-30T06:29:54.394ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22462025-04-30T06:29:54.395ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0fcc18b-e7e5-4f10-90b9-8463f8981140", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22472025-04-30T06:29:54.395ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
22482025-04-30T06:29:54.397ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1405ffe2-c11f-472b-ac46-e312436897b5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22492025-04-30T06:29:54.397ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
22502025-04-30T06:29:54.398ZTRACtest_image_promotion (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22512025-04-30T06:29:54.398ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22522025-04-30T06:29:54.398ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22532025-04-30T06:29:54.398ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22542025-04-30T06:29:54.399ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22552025-04-30T06:29:54.402ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bc7dfca-1da6-443e-a147-d5a6858948bc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22562025-04-30T06:29:54.403ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
22572025-04-30T06:29:54.403ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22582025-04-30T06:29:54.403ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22592025-04-30T06:29:54.403ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22602025-04-30T06:29:54.404ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22612025-04-30T06:29:54.407ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30c98a7a-b45a-4c3a-94e7-057d57826c58", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22622025-04-30T06:29:54.407ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
22632025-04-30T06:29:54.407ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22642025-04-30T06:29:54.407ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22652025-04-30T06:29:54.408ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22662025-04-30T06:29:54.410ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22672025-04-30T06:29:54.423ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "566b38ca-3e67-41ec-8ea0-3f9a3aa1de1d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22682025-04-30T06:29:54.424ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
22692025-04-30T06:29:54.424ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22702025-04-30T06:29:54.436ZDEBGtest_image_promotion (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22712025-04-30T06:29:54.436ZTRACtest_image_promotion (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22722025-04-30T06:29:54.436ZTRACtest_image_promotion (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22732025-04-30T06:29:54.436ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22742025-04-30T06:29:54.436ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22752025-04-30T06:29:54.436ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22762025-04-30T06:29:54.437ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22772025-04-30T06:29:54.439ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf76be8f-2d70-4470-b9c0-914baa5a8538", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22782025-04-30T06:29:54.440ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
22792025-04-30T06:29:54.440ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22802025-04-30T06:29:54.440ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22812025-04-30T06:29:54.440ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22822025-04-30T06:29:54.440ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22832025-04-30T06:29:54.443ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22842025-04-30T06:29:54.443ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22852025-04-30T06:29:54.443ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22862025-04-30T06:29:54.443ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22872025-04-30T06:29:54.444ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1222564-3f49-43a9-a087-0c9ac3330018", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22882025-04-30T06:29:54.444ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
22892025-04-30T06:29:54.446ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22902025-04-30T06:29:54.456ZDEBGtest_image_promotion (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22912025-04-30T06:29:54.456ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22922025-04-30T06:29:54.456ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
22932025-04-30T06:29:54.456ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22942025-04-30T06:29:54.457ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1b496aa-ca0c-4ad5-aeb0-8f087b4039db", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22952025-04-30T06:29:54.458ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
22962025-04-30T06:29:54.461ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebf68ecf-c4fe-4c70-9f43-e4859b462378", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22972025-04-30T06:29:54.461ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
22982025-04-30T06:29:54.464ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "03dd7590-464a-4200-8c0c-0c3fe8ac9d87", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
22992025-04-30T06:29:54.464ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
23002025-04-30T06:29:54.465ZDEBGtest_image_promotion (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: bf77b0f7-839e-436d-8370-ef5ca8a0ebae, kind: Service { id: d9b51ab0-9dce-44da-9764-8fd37f3b0498 }, name: Name("nexus-d9b51ab0-9dce-44da-9764-8fd37f3b0498"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 3d9a262e-f7ce-40cb-9f2f-7421ff0f6ec6, kind: Service { id: bc263bb9-670b-4189-802a-5cfd0b1be255 }, name: Name("external-dns-bc263bb9-670b-4189-802a-5cfd0b1be255"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
23012025-04-30T06:29:54.465ZTRACtest_image_promotion (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
23022025-04-30T06:29:54.465ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23032025-04-30T06:29:54.465ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23042025-04-30T06:29:54.465ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
23052025-04-30T06:29:54.465ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23062025-04-30T06:29:54.468ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94563cbd-506b-4586-addb-693140effe8b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23072025-04-30T06:29:54.469ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
23082025-04-30T06:29:54.469ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23092025-04-30T06:29:54.473ZDEBGtest_image_promotion (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
23102025-04-30T06:29:54.473ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23112025-04-30T06:29:54.473ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
23122025-04-30T06:29:54.473ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23132025-04-30T06:29:54.474ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1899749c-ab34-471b-a7e6-f8ff613d0aff", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23142025-04-30T06:29:54.475ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
23152025-04-30T06:29:54.476ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f67e6ca-b52a-409e-b70c-b51f18d87917", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23162025-04-30T06:29:54.476ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
23172025-04-30T06:29:54.476ZTRACtest_image_promotion (InternalApi): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:29:53.380695Z, allowed_ips: None }
23182025-04-30T06:29:54.476ZDEBGtest_image_promotion (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
23192025-04-30T06:29:54.476ZDEBGtest_image_promotion (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 3d9a262e-f7ce-40cb-9f2f-7421ff0f6ec6, kind: Service { id: bc263bb9-670b-4189-802a-5cfd0b1be255 }, name: Name("external-dns-bc263bb9-670b-4189-802a-5cfd0b1be255"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: bf77b0f7-839e-436d-8370-ef5ca8a0ebae, kind: Service { id: d9b51ab0-9dce-44da-9764-8fd37f3b0498 }, name: Name("nexus-d9b51ab0-9dce-44da-9764-8fd37f3b0498"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
23202025-04-30T06:29:54.476ZDEBGtest_image_promotion (InternalApi): resolved 2 rules for sleds
23212025-04-30T06:29:54.478ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f937b8d8-22b2-46b3-b902-9851c610b594", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23222025-04-30T06:29:54.478ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
23232025-04-30T06:29:54.494ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "caee09d4-64da-4469-9e7c-05111b96c551", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23242025-04-30T06:29:54.494ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
23252025-04-30T06:29:54.495ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "679d1e7e-6b7a-4251-ae38-0e43e3670dd0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23262025-04-30T06:29:54.495ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
23272025-04-30T06:29:54.497ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cba41b4-f26d-4b81-81ab-fc9d0e55a330", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23282025-04-30T06:29:54.497ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
23292025-04-30T06:29:54.499ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0890e1ee-515e-485e-a1fd-de9b23d8e113", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23302025-04-30T06:29:54.499ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
23312025-04-30T06:29:54.501ZDEBGtest_image_promotion (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:29:46.519511Z, time_modified: 2025-04-30T06:29:46.519511Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(60070), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(41286) }]
23322025-04-30T06:29:54.501ZDEBGtest_image_promotion (InternalApi): sending firewall rules to sled agents
23332025-04-30T06:29:54.501ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23342025-04-30T06:29:54.501ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
23352025-04-30T06:29:54.503ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23362025-04-30T06:29:54.505ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc95ea04-34e7-4551-a3c5-ca3ee2ba139e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23372025-04-30T06:29:54.505ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
23382025-04-30T06:29:54.508ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b40b6b9-221b-4559-b863-0557e40d2aa1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23392025-04-30T06:29:54.509ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
23402025-04-30T06:29:54.510ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
23412025-04-30T06:29:54.510ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23422025-04-30T06:29:54.510ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23432025-04-30T06:29:54.510ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
23442025-04-30T06:29:54.511ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23452025-04-30T06:29:54.511ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e436c2a-8afe-45b1-8bbd-acca348f4524", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23462025-04-30T06:29:54.512ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
23472025-04-30T06:29:54.514ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8c96668-a662-49e3-8194-c99ffc62e1b0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23482025-04-30T06:29:54.515ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
23492025-04-30T06:29:54.516ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23502025-04-30T06:29:54.529ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
23512025-04-30T06:29:54.576ZDEBGtest_image_promotion (InternalApi): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:60070/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23522025-04-30T06:29:54.578ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:58871 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23532025-04-30T06:29:54.579ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = PUT remote_addr = [::1]:58871 req_id = 2551c014-2bf0-4fde-8e84-b4ce708d25b7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23542025-04-30T06:29:54.579ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81086146-2865-47a4-a9da-82da658281b9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23552025-04-30T06:29:54.579ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
23562025-04-30T06:29:54.580ZINFOtest_image_promotion (dropshot): request completed latency_us = 1307 local_addr = [::1]:60070 method = PUT remote_addr = [::1]:58871 req_id = 2551c014-2bf0-4fde-8e84-b4ce708d25b7 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23572025-04-30T06:29:54.581ZDEBGtest_image_promotion (InternalApi): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:60070/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "2551c014-2bf0-4fde-8e84-b4ce708d25b7", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
23582025-04-30T06:29:54.582ZINFOtest_image_promotion (InternalApi): updated firewall rules on 1 sleds
23592025-04-30T06:29:54.582ZINFOtest_image_promotion (nexus): plumbed initial IP allowlist
23602025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = / versions = all
23612025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /access versions = all
23622025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /assets/{path} versions = all
23632025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /device/auth versions = all
23642025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /device/confirm versions = all
23652025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /device/success versions = all
23662025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /device/token versions = all
23672025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /device/verify versions = all
23682025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /experimental/v1/probes versions = all
23692025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /experimental/v1/probes versions = all
23702025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /experimental/v1/probes/{probe} versions = all
23712025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /experimental/v1/probes/{probe} versions = all
23722025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /experimental/v1/system/support-bundles versions = all
23732025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /experimental/v1/system/support-bundles versions = all
23742025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23752025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23762025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23772025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23782025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
23792025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
23802025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
23812025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /images versions = all
23822025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /login versions = all
23832025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /login/{silo_name}/local versions = all
23842025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
23852025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
23862025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
23872025-04-30T06:29:54.670ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /lookup/{path} versions = all
23882025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /projects/{path} versions = all
23892025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /projects-new versions = all
23902025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /settings/{path} versions = all
23912025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /system/{path} versions = all
23922025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /utilization versions = all
23932025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/affinity-groups versions = all
23942025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/affinity-groups versions = all
23952025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
23962025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
23972025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
23982025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
23992025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24002025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24012025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24022025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/anti-affinity-groups versions = all
24032025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/anti-affinity-groups versions = all
24042025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24052025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24062025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24072025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
24082025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24092025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24102025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24112025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/certificates versions = all
24122025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/certificates versions = all
24132025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/certificates/{certificate} versions = all
24142025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/certificates/{certificate} versions = all
24152025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/disks versions = all
24162025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/disks versions = all
24172025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/disks/{disk} versions = all
24182025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/disks/{disk} versions = all
24192025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/disks/{disk}/bulk-write versions = all
24202025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
24212025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
24222025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/disks/{disk}/finalize versions = all
24232025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
24242025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/floating-ips versions = all
24252025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/floating-ips versions = all
24262025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
24272025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/floating-ips/{floating_ip} versions = all
24282025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
24292025-04-30T06:29:54.671ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
24302025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
24312025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/groups versions = all
24322025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/groups/{group_id} versions = all
24332025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/images versions = all
24342025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/images versions = all
24352025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/images/{image} versions = all
24362025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/images/{image} versions = all
24372025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/images/{image}/demote versions = all
24382025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/images/{image}/promote versions = all
24392025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/instances versions = all
24402025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/instances versions = all
24412025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/instances/{instance} versions = all
24422025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/instances/{instance} versions = all
24432025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/instances/{instance} versions = all
24442025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
24452025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
24462025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/instances/{instance}/disks versions = all
24472025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/instances/{instance}/disks/attach versions = all
24482025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/instances/{instance}/disks/detach versions = all
24492025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/instances/{instance}/external-ips versions = all
24502025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24512025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24522025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/instances/{instance}/reboot versions = all
24532025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/instances/{instance}/serial-console versions = all
24542025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
24552025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
24562025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/instances/{instance}/start versions = all
24572025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/instances/{instance}/stop versions = all
24582025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/internet-gateway-ip-addresses versions = all
24592025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/internet-gateway-ip-addresses versions = all
24602025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
24612025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/internet-gateway-ip-pools versions = all
24622025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/internet-gateway-ip-pools versions = all
24632025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
24642025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/internet-gateways versions = all
24652025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/internet-gateways versions = all
24662025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
24672025-04-30T06:29:54.672ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/internet-gateways/{gateway} versions = all
24682025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/ip-pools versions = all
24692025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/ip-pools/{pool} versions = all
24702025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/login/{silo_name}/local versions = all
24712025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/logout versions = all
24722025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/me versions = all
24732025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/me/groups versions = all
24742025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/me/ssh-keys versions = all
24752025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/me/ssh-keys versions = all
24762025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
24772025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
24782025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/metrics/{metric_name} versions = all
24792025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/network-interfaces versions = all
24802025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/network-interfaces versions = all
24812025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/network-interfaces/{interface} versions = all
24822025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/network-interfaces/{interface} versions = all
24832025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/network-interfaces/{interface} versions = all
24842025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/ping versions = all
24852025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/policy versions = all
24862025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/policy versions = all
24872025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/projects versions = all
24882025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/projects versions = all
24892025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/projects/{project} versions = all
24902025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/projects/{project} versions = all
24912025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/projects/{project} versions = all
24922025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/projects/{project}/policy versions = all
24932025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/projects/{project}/policy versions = all
24942025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/snapshots versions = all
24952025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/snapshots versions = all
24962025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/snapshots/{snapshot} versions = all
24972025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/snapshots/{snapshot} versions = all
24982025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/disks versions = all
24992025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
25002025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
25012025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/racks versions = all
25022025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
25032025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/sleds versions = all
25042025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/hardware/sleds versions = all
25052025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
25062025-04-30T06:29:54.673ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
25072025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
25082025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
25092025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
25102025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/switch-port versions = all
25112025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25122025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25132025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
25142025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
25152025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
25162025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/switches versions = all
25172025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
25182025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/identity-providers versions = all
25192025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/identity-providers/local/users versions = all
25202025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
25212025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
25222025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/identity-providers/saml versions = all
25232025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
25242025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/ip-pools versions = all
25252025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/ip-pools versions = all
25262025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
25272025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/ip-pools/{pool} versions = all
25282025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/system/ip-pools/{pool} versions = all
25292025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
25302025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
25312025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
25322025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
25332025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
25342025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25352025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25362025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
25372025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/ip-pools-service versions = all
25382025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/ip-pools-service/ranges versions = all
25392025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
25402025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
25412025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/metrics/{metric_name} versions = all
25422025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/address-lot versions = all
25432025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/networking/address-lot versions = all
25442025-04-30T06:29:54.674ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
25452025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
25462025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/allow-list versions = all
25472025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/system/networking/allow-list versions = all
25482025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/networking/bfd-disable versions = all
25492025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/networking/bfd-enable versions = all
25502025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/bfd-status versions = all
25512025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/system/networking/bgp versions = all
25522025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/bgp versions = all
25532025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/networking/bgp versions = all
25542025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/bgp-announce-set versions = all
25552025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
25562025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
25572025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
25582025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/bgp-exported versions = all
25592025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/bgp-message-history versions = all
25602025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
25612025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/bgp-status versions = all
25622025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/loopback-address versions = all
25632025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/networking/loopback-address versions = all
25642025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
25652025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
25662025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/switch-port-settings versions = all
25672025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/networking/switch-port-settings versions = all
25682025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
25692025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/policy versions = all
25702025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/system/policy versions = all
25712025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/roles versions = all
25722025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/roles/{role_name} versions = all
25732025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/silo-quotas versions = all
25742025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/silos versions = all
25752025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/silos versions = all
25762025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/system/silos/{silo} versions = all
25772025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/silos/{silo} versions = all
25782025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
25792025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/silos/{silo}/policy versions = all
25802025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/system/silos/{silo}/policy versions = all
25812025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/silos/{silo}/quotas versions = all
25822025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
25832025-04-30T06:29:54.675ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/system/timeseries/query versions = all
25842025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/timeseries/schemas versions = all
25852025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/system/update/repository versions = all
25862025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/update/repository/{system_version} versions = all
25872025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/update/target-release versions = all
25882025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/system/update/target-release versions = all
25892025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/users versions = all
25902025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/users/{user_id} versions = all
25912025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/users-builtin versions = all
25922025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/users-builtin/{user} versions = all
25932025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/utilization/silos versions = all
25942025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/system/utilization/silos/{silo} versions = all
25952025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/timeseries/query versions = all
25962025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/users versions = all
25972025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/utilization versions = all
25982025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/vpc-firewall-rules versions = all
25992025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/vpc-firewall-rules versions = all
26002025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/vpc-router-routes versions = all
26012025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/vpc-router-routes versions = all
26022025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
26032025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/vpc-router-routes/{route} versions = all
26042025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/vpc-router-routes/{route} versions = all
26052025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/vpc-routers versions = all
26062025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/vpc-routers versions = all
26072025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/vpc-routers/{router} versions = all
26082025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/vpc-routers/{router} versions = all
26092025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/vpc-routers/{router} versions = all
26102025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/vpc-subnets versions = all
26112025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/vpc-subnets versions = all
26122025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
26132025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/vpc-subnets/{subnet} versions = all
26142025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
26152025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
26162025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/vpcs versions = all
26172025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/vpcs versions = all
26182025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/vpcs/{vpc} versions = all
26192025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/vpcs/{vpc} versions = all
26202025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/vpcs/{vpc} versions = all
26212025-04-30T06:29:54.676ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/webhooks/deliveries versions = all
26222025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
26232025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/webhooks/event-classes versions = all
26242025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/webhooks/receivers versions = all
26252025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/webhooks/receivers versions = all
26262025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
26272025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
26282025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
26292025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
26302025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
26312025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
26322025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = GET path = /v1/webhooks/secrets versions = all
26332025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = POST path = /v1/webhooks/secrets versions = all
26342025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): registered endpoint local_addr = 127.0.0.1:62043 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
26352025-04-30T06:29:54.677ZINFOtest_image_promotion (dropshot_external): listening local_addr = 127.0.0.1:62043
26362025-04-30T06:29:54.677ZDEBGtest_image_promotion (dropshot_external): successfully registered DTrace USDT probes local_addr = 127.0.0.1:62043
26372025-04-30T06:29:54.727ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = / versions = all
26382025-04-30T06:29:54.727ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /access versions = all
26392025-04-30T06:29:54.727ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /assets/{path} versions = all
26402025-04-30T06:29:54.727ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /device/auth versions = all
26412025-04-30T06:29:54.727ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /device/confirm versions = all
26422025-04-30T06:29:54.727ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /device/success versions = all
26432025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /device/token versions = all
26442025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /device/verify versions = all
26452025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /experimental/v1/probes versions = all
26462025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /experimental/v1/probes versions = all
26472025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /experimental/v1/probes/{probe} versions = all
26482025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /experimental/v1/probes/{probe} versions = all
26492025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /experimental/v1/system/support-bundles versions = all
26502025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /experimental/v1/system/support-bundles versions = all
26512025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26522025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26532025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26542025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26552025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26562025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26572025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
26582025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /images versions = all
26592025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /login versions = all
26602025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /login/{silo_name}/local versions = all
26612025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
26622025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
26632025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
26642025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /lookup/{path} versions = all
26652025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /projects/{path} versions = all
26662025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /projects-new versions = all
26672025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /settings/{path} versions = all
26682025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /system/{path} versions = all
26692025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /utilization versions = all
26702025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/affinity-groups versions = all
26712025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/affinity-groups versions = all
26722025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
26732025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
26742025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
26752025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
26762025-04-30T06:29:54.728ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26772025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26782025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26792025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/anti-affinity-groups versions = all
26802025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/anti-affinity-groups versions = all
26812025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26822025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26832025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26842025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
26852025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26862025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26872025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26882025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/certificates versions = all
26892025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/certificates versions = all
26902025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/certificates/{certificate} versions = all
26912025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/certificates/{certificate} versions = all
26922025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/disks versions = all
26932025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/disks versions = all
26942025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/disks/{disk} versions = all
26952025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/disks/{disk} versions = all
26962025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/disks/{disk}/bulk-write versions = all
26972025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
26982025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
26992025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/disks/{disk}/finalize versions = all
27002025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
27012025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/floating-ips versions = all
27022025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/floating-ips versions = all
27032025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
27042025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/floating-ips/{floating_ip} versions = all
27052025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
27062025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
27072025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
27082025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/groups versions = all
27092025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/groups/{group_id} versions = all
27102025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/images versions = all
27112025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/images versions = all
27122025-04-30T06:29:54.729ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/images/{image} versions = all
27132025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/images/{image} versions = all
27142025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/images/{image}/demote versions = all
27152025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/images/{image}/promote versions = all
27162025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/instances versions = all
27172025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/instances versions = all
27182025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/instances/{instance} versions = all
27192025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/instances/{instance} versions = all
27202025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/instances/{instance} versions = all
27212025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
27222025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
27232025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/instances/{instance}/disks versions = all
27242025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/instances/{instance}/disks/attach versions = all
27252025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/instances/{instance}/disks/detach versions = all
27262025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/instances/{instance}/external-ips versions = all
27272025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27282025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27292025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/instances/{instance}/reboot versions = all
27302025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/instances/{instance}/serial-console versions = all
27312025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
27322025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
27332025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/instances/{instance}/start versions = all
27342025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/instances/{instance}/stop versions = all
27352025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/internet-gateway-ip-addresses versions = all
27362025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/internet-gateway-ip-addresses versions = all
27372025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
27382025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/internet-gateway-ip-pools versions = all
27392025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/internet-gateway-ip-pools versions = all
27402025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
27412025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/internet-gateways versions = all
27422025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/internet-gateways versions = all
27432025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
27442025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/internet-gateways/{gateway} versions = all
27452025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/ip-pools versions = all
27462025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/ip-pools/{pool} versions = all
27472025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/login/{silo_name}/local versions = all
27482025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/logout versions = all
27492025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/me versions = all
27502025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/me/groups versions = all
27512025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/me/ssh-keys versions = all
27522025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/me/ssh-keys versions = all
27532025-04-30T06:29:54.730ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
27542025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
27552025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/metrics/{metric_name} versions = all
27562025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/network-interfaces versions = all
27572025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/network-interfaces versions = all
27582025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/network-interfaces/{interface} versions = all
27592025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/network-interfaces/{interface} versions = all
27602025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/network-interfaces/{interface} versions = all
27612025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/ping versions = all
27622025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/policy versions = all
27632025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/policy versions = all
27642025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/projects versions = all
27652025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/projects versions = all
27662025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/projects/{project} versions = all
27672025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/projects/{project} versions = all
27682025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/projects/{project} versions = all
27692025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/projects/{project}/policy versions = all
27702025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/projects/{project}/policy versions = all
27712025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/snapshots versions = all
27722025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/snapshots versions = all
27732025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/snapshots/{snapshot} versions = all
27742025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/snapshots/{snapshot} versions = all
27752025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/disks versions = all
27762025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
27772025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
27782025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/racks versions = all
27792025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
27802025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/sleds versions = all
27812025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/hardware/sleds versions = all
27822025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
27832025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
27842025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
27852025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
27862025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
27872025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/switch-port versions = all
27882025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27892025-04-30T06:29:54.731ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27902025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
27912025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
27922025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
27932025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/switches versions = all
27942025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
27952025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/identity-providers versions = all
27962025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/identity-providers/local/users versions = all
27972025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
27982025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
27992025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/identity-providers/saml versions = all
28002025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
28012025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/ip-pools versions = all
28022025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/ip-pools versions = all
28032025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
28042025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/ip-pools/{pool} versions = all
28052025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/system/ip-pools/{pool} versions = all
28062025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
28072025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
28082025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
28092025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
28102025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
28112025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28122025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28132025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
28142025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/ip-pools-service versions = all
28152025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/ip-pools-service/ranges versions = all
28162025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
28172025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
28182025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/metrics/{metric_name} versions = all
28192025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/address-lot versions = all
28202025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/networking/address-lot versions = all
28212025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
28222025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
28232025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/allow-list versions = all
28242025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/system/networking/allow-list versions = all
28252025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/networking/bfd-disable versions = all
28262025-04-30T06:29:54.732ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/networking/bfd-enable versions = all
28272025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/bfd-status versions = all
28282025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/system/networking/bgp versions = all
28292025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/bgp versions = all
28302025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/networking/bgp versions = all
28312025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/bgp-announce-set versions = all
28322025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
28332025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
28342025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
28352025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/bgp-exported versions = all
28362025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/bgp-message-history versions = all
28372025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
28382025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/bgp-status versions = all
28392025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/loopback-address versions = all
28402025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/networking/loopback-address versions = all
28412025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
28422025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
28432025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/switch-port-settings versions = all
28442025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/networking/switch-port-settings versions = all
28452025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
28462025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/policy versions = all
28472025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/system/policy versions = all
28482025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/roles versions = all
28492025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/roles/{role_name} versions = all
28502025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/silo-quotas versions = all
28512025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/silos versions = all
28522025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/silos versions = all
28532025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/system/silos/{silo} versions = all
28542025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/silos/{silo} versions = all
28552025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
28562025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/silos/{silo}/policy versions = all
28572025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/system/silos/{silo}/policy versions = all
28582025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/silos/{silo}/quotas versions = all
28592025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
28602025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/system/timeseries/query versions = all
28612025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/timeseries/schemas versions = all
28622025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/system/update/repository versions = all
28632025-04-30T06:29:54.733ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/update/repository/{system_version} versions = all
28642025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/update/target-release versions = all
28652025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/system/update/target-release versions = all
28662025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/users versions = all
28672025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/users/{user_id} versions = all
28682025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/users-builtin versions = all
28692025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/users-builtin/{user} versions = all
28702025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/utilization/silos versions = all
28712025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/system/utilization/silos/{silo} versions = all
28722025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/timeseries/query versions = all
28732025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/users versions = all
28742025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/utilization versions = all
28752025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/vpc-firewall-rules versions = all
28762025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/vpc-firewall-rules versions = all
28772025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/vpc-router-routes versions = all
28782025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/vpc-router-routes versions = all
28792025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
28802025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/vpc-router-routes/{route} versions = all
28812025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/vpc-router-routes/{route} versions = all
28822025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/vpc-routers versions = all
28832025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/vpc-routers versions = all
28842025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/vpc-routers/{router} versions = all
28852025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/vpc-routers/{router} versions = all
28862025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/vpc-routers/{router} versions = all
28872025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/vpc-subnets versions = all
28882025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/vpc-subnets versions = all
28892025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
28902025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/vpc-subnets/{subnet} versions = all
28912025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
28922025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
28932025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/vpcs versions = all
28942025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/vpcs versions = all
28952025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/vpcs/{vpc} versions = all
28962025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/vpcs/{vpc} versions = all
28972025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/vpcs/{vpc} versions = all
28982025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/webhooks/deliveries versions = all
28992025-04-30T06:29:54.734ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
29002025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/webhooks/event-classes versions = all
29012025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/webhooks/receivers versions = all
29022025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/webhooks/receivers versions = all
29032025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
29042025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
29052025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
29062025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
29072025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
29082025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
29092025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = GET path = /v1/webhooks/secrets versions = all
29102025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = POST path = /v1/webhooks/secrets versions = all
29112025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:54789 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
29122025-04-30T06:29:54.735ZINFOtest_image_promotion (dropshot_external_techport): listening local_addr = 127.0.0.1:54789
29132025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = 127.0.0.1:54789
29142025-04-30T06:29:54.735ZDEBGtest_image_promotion (nexus-producer-server): registered DTrace probes
29152025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = 127.0.0.1:60931 method = GET path = /{producer_id} versions = all
29162025-04-30T06:29:54.735ZINFOtest_image_promotion (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 127.0.0.1:60931
29172025-04-30T06:29:54.735ZDEBGtest_image_promotion (dropshot): successfully registered DTrace USDT probes local_addr = 127.0.0.1:60931
29182025-04-30T06:29:54.735ZDEBGtest_image_promotion (nexus-producer-server): Requested any available port, Dropshot server has been bound to 127.0.0.1:60931
29192025-04-30T06:29:54.735ZDEBGtest_image_promotion (nexus-producer-server): Nexus IP provided explicitly, will use it addr = 127.0.0.1:45630
29202025-04-30T06:29:54.735ZDEBGtest_image_promotion (nexus-producer-server): starting producer registration task
29212025-04-30T06:29:54.735ZINFOtest_image_promotion (nexus-producer-server): starting oximeter metric producer server address = 127.0.0.1:60931 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
29222025-04-30T06:29:54.735ZDEBGtest_image_promotion (init_with_steps): Running step start_oximeter
29232025-04-30T06:29:54.735ZDEBGtest_image_promotion: Starting Oximeter
29242025-04-30T06:29:54.735ZDEBGtest_image_promotion (oximeter): registered DTrace probes
29252025-04-30T06:29:54.735ZINFOtest_image_promotion (oximeter): starting oximeter server file = oximeter/collector/src/lib.rs:236
29262025-04-30T06:29:54.735ZDEBGtest_image_promotion (oximeter): creating ClickHouse client
29272025-04-30T06:29:54.735ZDEBGtest_image_promotion (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
29282025-04-30T06:29:54.735ZDEBGtest_image_promotion (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0
29292025-04-30T06:29:54.735ZDEBGtest_image_promotion (producer-registration-task): will register with Nexus at 127.0.0.1:45630
29302025-04-30T06:29:54.790ZDEBGtest_image_promotion (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:45630/metrics/producers
29312025-04-30T06:29:54.794ZINFOtest_image_promotion (dropshot_internal): accepted connection local_addr = 127.0.0.1:45630 remote_addr = 127.0.0.1:47332
29322025-04-30T06:29:54.805ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:47332 req_id = b4f39e4a-2d71-4770-8f59-bec17802f152 uri = /metrics/producers
29332025-04-30T06:29:54.805ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "935fc495-6ca0-4008-a122-c3d91607fcf3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
29342025-04-30T06:29:54.806ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
29352025-04-30T06:29:54.807ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:47332 req_id = b4f39e4a-2d71-4770-8f59-bec17802f152 resource = Database uri = /metrics/producers
29362025-04-30T06:29:54.807ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:47332 req_id = b4f39e4a-2d71-4770-8f59-bec17802f152 roles = RoleSet { roles: {} } uri = /metrics/producers
29372025-04-30T06:29:54.812ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:47332 req_id = b4f39e4a-2d71-4770-8f59-bec17802f152 resource = Database result = Ok(()) uri = /metrics/producers
29382025-04-30T06:29:54.813ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29392025-04-30T06:29:54.815ZTRACtest_image_promotion (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: f6 74 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 .t..........._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:50888 req_id = a5a0f090-b5ef-441f-a7be-6126508a3d57
29402025-04-30T06:29:54.816ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63092,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50888 req_id = a5a0f090-b5ef-441f-a7be-6126508a3d57
29412025-04-30T06:29:54.816ZDEBGtest_image_promotion (store): query key key = _crucible-pantry._tcp
29422025-04-30T06:29:54.817ZDEBGtest_image_promotion (store): query key key = a69550cd-17cf-4769-b40b-06fad3e3da70.host
29432025-04-30T06:29:54.817ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("a69550cd-17cf-4769-b40b-06fad3e3da70.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50888 query = LowerQuery { name: LowerName(Name("_crucible-pantry._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65415, target: Name("a69550cd-17cf-4769-b40b-06fad3e3da70.host.control-plane.oxide.internal") })) }] req_id = a5a0f090-b5ef-441f-a7be-6126508a3d57
29442025-04-30T06:29:54.823ZTRACtest_image_promotion (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 82 55 01 00 00 01 00 00 00 00 00 01 24 61 36 39 .U..........$a69\n0010: 35 35 30 63 64 2d 31 37 63 66 2d 34 37 36 39 2d 550cd-17cf-4769-\n0020: 62 34 30 62 2d 30 36 66 61 64 33 65 33 64 61 37 b40b-06fad3e3da7\n0030: 30 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 0.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:62756 req_id = 79b767b7-de80-441b-a7b0-97d7b6373898
29452025-04-30T06:29:54.824ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33365,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("a69550cd-17cf-4769-b40b-06fad3e3da70.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("a69550cd-17cf-4769-b40b-06fad3e3da70.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 97,\n 54,\n 57,\n 53,\n 53,\n 48,\n 99,\n 100,\n 45,\n 49,\n 55,\n 99,\n 102,\n 45,\n 52,\n 55,\n 54,\n 57,\n 45,\n 98,\n 52,\n 48,\n 98,\n 45,\n 48,\n 54,\n 102,\n 97,\n 100,\n 51,\n 101,\n 51,\n 100,\n 97,\n 55,\n 48,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62756 req_id = 79b767b7-de80-441b-a7b0-97d7b6373898
29462025-04-30T06:29:54.824ZDEBGtest_image_promotion (store): query key key = a69550cd-17cf-4769-b40b-06fad3e3da70.host
29472025-04-30T06:29:54.824ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62756 query = LowerQuery { name: LowerName(Name("a69550cd-17cf-4769-b40b-06fad3e3da70.host.control-plane.oxide.internal.")), original: Query { name: Name("a69550cd-17cf-4769-b40b-06fad3e3da70.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("a69550cd-17cf-4769-b40b-06fad3e3da70.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 79b767b7-de80-441b-a7b0-97d7b6373898
29482025-04-30T06:29:54.825ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e983aa88-cab9-443e-93d5-d93fe973a5e3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
29492025-04-30T06:29:54.825ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
29502025-04-30T06:29:54.893ZINFOtest_image_promotion (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 88258 local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:47332 req_id = b4f39e4a-2d71-4770-8f59-bec17802f152 response_code = 503 uri = /metrics/producers
29512025-04-30T06:29:54.893ZWARNtest_image_promotion (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = a716977f-f822-467e-b942-62720757c880
29522025-04-30T06:29:54.894ZDEBGtest_image_promotion (oximeter-agent): oximeter database does not exist, creating collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
29532025-04-30T06:29:54.894ZINFOtest_image_promotion (clickhouse-client): reading db version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:794 id = a716977f-f822-467e-b942-62720757c880
29542025-04-30T06:29:54.894ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:47345 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29552025-04-30T06:29:54.895ZDEBGtest_image_promotion (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "b4f39e4a-2d71-4770-8f59-bec17802f152", "content-length": "133", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
29562025-04-30T06:29:54.896ZWARNtest_image_promotion (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 163.18797ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"b4f39e4a-2d71-4770-8f59-bec17802f152\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:29:54 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"b4f39e4a-2d71-4770-8f59-bec17802f152\\" }" file = oximeter/producer/src/lib.rs:371
29572025-04-30T06:29:54.897ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29582025-04-30T06:29:54.898ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:47345 req_id = d71e9dd7-e208-460a-8275-dbf2bdfdca48 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29592025-04-30T06:29:54.898ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 654 local_addr = [::1]:65415 method = GET remote_addr = [::1]:47345 req_id = d71e9dd7-e208-460a-8275-dbf2bdfdca48 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29602025-04-30T06:29:54.899ZWARNtest_image_promotion (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = a716977f-f822-467e-b942-62720757c880
29612025-04-30T06:29:54.899ZINFOtest_image_promotion (clickhouse-client): read oximeter database version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:799 id = a716977f-f822-467e-b942-62720757c880 version = 0
29622025-04-30T06:29:54.899ZINFOtest_image_promotion (clickhouse-client): initializing oximeter schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:804 id = a716977f-f822-467e-b942-62720757c880
29632025-04-30T06:29:54.899ZDEBGtest_image_promotion (clickhouse-client): initializing ClickHouse database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
29642025-04-30T06:29:54.899ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7506a60c-588f-48ef-a3b7-fc64a1cbbdc5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:54 GMT"} })
29652025-04-30T06:29:54.899ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29662025-04-30T06:29:54.899ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29672025-04-30T06:29:54.899ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29682025-04-30T06:29:54.899ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29692025-04-30T06:29:54.900ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29702025-04-30T06:29:55.303ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = CREATE DATABASE IF NOT EXISTS oximeter
29712025-04-30T06:29:55.304ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:58816 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29722025-04-30T06:29:55.304ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:61485 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29732025-04-30T06:29:55.304ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:54293 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29742025-04-30T06:29:55.304ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:39554 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29752025-04-30T06:29:55.304ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:57442 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29762025-04-30T06:29:55.304ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:57431 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29772025-04-30T06:29:55.304ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:37291 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29782025-04-30T06:29:55.305ZDEBGtest_image_promotion (producer-registration-task): will register with Nexus at 127.0.0.1:45630
29792025-04-30T06:29:55.366ZDEBGtest_image_promotion (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:45630/metrics/producers
29802025-04-30T06:29:55.367ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29812025-04-30T06:29:55.372ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
29822025-04-30T06:29:55.372ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29832025-04-30T06:29:55.372ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29842025-04-30T06:29:55.372ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29852025-04-30T06:29:55.372ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29862025-04-30T06:29:55.373ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29872025-04-30T06:29:55.374ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\n/* The version table contains metadata about the `oximeter` database */\nCREATE TABLE IF NOT EXISTS oximeter.version\n(\n value UInt64,\n timestamp DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (value, timestamp)
29882025-04-30T06:29:55.375ZINFOtest_image_promotion (dropshot_internal): accepted connection local_addr = 127.0.0.1:45630 remote_addr = 127.0.0.1:57426
29892025-04-30T06:29:55.375ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:58816 req_id = f26e6b88-7f1d-415c-9284-521702ecdb56 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29902025-04-30T06:29:55.375ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:61485 req_id = 9ea85e98-aa64-40c1-adb0-5956080c997d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29912025-04-30T06:29:55.376ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:54293 req_id = c531b247-322d-4e3e-93c4-ecd44b2f5256 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29922025-04-30T06:29:55.376ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:39554 req_id = 41a3d33d-b594-4b88-8b15-b318ca41a40d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29932025-04-30T06:29:55.376ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:57442 req_id = 2a8b75d1-0722-430f-8cbb-bacdcd28e12e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29942025-04-30T06:29:55.376ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:57431 req_id = a1fbf360-921b-4d64-ba9e-23f918e93651 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29952025-04-30T06:29:55.376ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:37291 req_id = 5592abdb-d682-47ea-a38b-c380ccbd0653 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29962025-04-30T06:29:55.377ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 1280 local_addr = [::1]:65415 method = GET remote_addr = [::1]:58816 req_id = f26e6b88-7f1d-415c-9284-521702ecdb56 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29972025-04-30T06:29:55.377ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 1213 local_addr = [::1]:65415 method = GET remote_addr = [::1]:61485 req_id = 9ea85e98-aa64-40c1-adb0-5956080c997d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29982025-04-30T06:29:55.377ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 1139 local_addr = [::1]:65415 method = GET remote_addr = [::1]:54293 req_id = c531b247-322d-4e3e-93c4-ecd44b2f5256 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29992025-04-30T06:29:55.377ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 1058 local_addr = [::1]:65415 method = GET remote_addr = [::1]:39554 req_id = 41a3d33d-b594-4b88-8b15-b318ca41a40d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30002025-04-30T06:29:55.377ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 841 local_addr = [::1]:65415 method = GET remote_addr = [::1]:57442 req_id = 2a8b75d1-0722-430f-8cbb-bacdcd28e12e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30012025-04-30T06:29:55.377ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 742 local_addr = [::1]:65415 method = GET remote_addr = [::1]:57431 req_id = a1fbf360-921b-4d64-ba9e-23f918e93651 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30022025-04-30T06:29:55.377ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 667 local_addr = [::1]:65415 method = GET remote_addr = [::1]:37291 req_id = 5592abdb-d682-47ea-a38b-c380ccbd0653 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30032025-04-30T06:29:55.379ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:57426 req_id = 0a2e90d9-f6dc-4915-af78-bfa28db8d545 uri = /metrics/producers
30042025-04-30T06:29:55.380ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:57426 req_id = 0a2e90d9-f6dc-4915-af78-bfa28db8d545 resource = Database uri = /metrics/producers
30052025-04-30T06:29:55.381ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:57426 req_id = 0a2e90d9-f6dc-4915-af78-bfa28db8d545 roles = RoleSet { roles: {} } uri = /metrics/producers
30062025-04-30T06:29:55.381ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:57426 req_id = 0a2e90d9-f6dc-4915-af78-bfa28db8d545 resource = Database result = Ok(()) uri = /metrics/producers
30072025-04-30T06:29:55.382ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30082025-04-30T06:29:55.387ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
30092025-04-30T06:29:55.387ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
30102025-04-30T06:29:55.387ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
30112025-04-30T06:29:55.388ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30122025-04-30T06:29:55.388ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\n/* The measurement tables contain all individual samples from each timeseries.\n *\n * Each table stores a single datum type, and otherwise contains nearly the same\n * structure. The primary sorting key is on the timeseries name, key, and then\n * timestamp, so that all timeseries from the same schema are grouped, followed\n * by all samples from the same timeseries.\n *\n * This reflects that one usually looks up the _key_ in one or more field table,\n * and then uses that to index quickly into the measurements tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Bool)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30132025-04-30T06:29:55.422ZINFOtest_image_promotion (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 43681 local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:57426 req_id = 0a2e90d9-f6dc-4915-af78-bfa28db8d545 response_code = 503 uri = /metrics/producers
30142025-04-30T06:29:55.423ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30152025-04-30T06:29:55.424ZDEBGtest_image_promotion (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "0a2e90d9-f6dc-4915-af78-bfa28db8d545", "content-length": "133", "date": "Wed, 30 Apr 2025 06:29:55 GMT"} })
30162025-04-30T06:29:55.425ZWARNtest_image_promotion (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 691.422303ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"0a2e90d9-f6dc-4915-af78-bfa28db8d545\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:29:55 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"0a2e90d9-f6dc-4915-af78-bfa28db8d545\\" }" file = oximeter/producer/src/lib.rs:371
30172025-04-30T06:29:55.436ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30182025-04-30T06:29:55.442ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30192025-04-30T06:29:55.449ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30202025-04-30T06:29:55.454ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30212025-04-30T06:29:55.507ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30222025-04-30T06:29:55.584ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
30232025-04-30T06:29:55.584ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
30242025-04-30T06:29:55.584ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30252025-04-30T06:29:55.585ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:45588 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30262025-04-30T06:29:55.588ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:45588 req_id = d214e376-48dc-4941-83d8-312803ac3c3f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
30272025-04-30T06:29:55.588ZINFOtest_image_promotion (dropshot): request completed latency_us = 852 local_addr = [::1]:60070 method = GET remote_addr = [::1]:45588 req_id = d214e376-48dc-4941-83d8-312803ac3c3f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
30282025-04-30T06:29:55.589ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30292025-04-30T06:29:55.589ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d214e376-48dc-4941-83d8-312803ac3c3f", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:29:55 GMT"} })
30302025-04-30T06:29:55.590ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
30312025-04-30T06:29:55.591ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
30322025-04-30T06:29:55.591ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
30332025-04-30T06:29:55.592ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
30342025-04-30T06:29:55.593ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
30352025-04-30T06:29:55.593ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
30362025-04-30T06:29:55.593ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
30372025-04-30T06:29:55.593ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30382025-04-30T06:29:55.594ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30392025-04-30T06:29:55.597ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30402025-04-30T06:29:55.597ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30412025-04-30T06:29:55.604ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
30422025-04-30T06:29:55.604ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
30432025-04-30T06:29:55.604ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
30442025-04-30T06:29:55.604ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30452025-04-30T06:29:55.609ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(String)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30462025-04-30T06:29:55.612ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bytes\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Right now we can't unambiguously record a nullable byte array.\n * Arrays cannot be nested in `Nullable()` types, and encoding the array as\n * a string isn't palatable for a few reasons.\n * See: https://github.com/oxidecomputer/omicron/issues/4551 for more\n * details.\n */\n datum Array(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30472025-04-30T06:29:55.615ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativei64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30482025-04-30T06:29:55.618ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativeu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30492025-04-30T06:29:55.621ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30502025-04-30T06:29:55.625ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30512025-04-30T06:29:55.628ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Array types cannot be Nullable, see\n * https://clickhouse.com/docs/en/sql-reference/data-types/nullable\n * for more details.\n *\n * This means we need to use empty arrays to indicate a missing value. This\n * is unfortunate, and at this point relies on the fact that an\n * `oximeter::Histogram` cannot have zero bins. If that changes, we'll need\n * to figure out another way to represent missing samples here.\n */\n bins Array(Int8),\n counts Array(UInt64),\n min Int8,\n max Int8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30522025-04-30T06:29:55.632ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt8),\n counts Array(UInt64),\n min UInt8,\n max UInt8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30532025-04-30T06:29:55.636ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int16),\n counts Array(UInt64),\n min Int16,\n max Int16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30542025-04-30T06:29:55.639ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt16),\n counts Array(UInt64),\n min UInt16,\n max UInt16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30552025-04-30T06:29:55.643ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int32),\n counts Array(UInt64),\n min Int32,\n max Int32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30562025-04-30T06:29:55.646ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt32),\n counts Array(UInt64),\n min UInt32,\n max UInt32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30572025-04-30T06:29:55.651ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int64),\n counts Array(UInt64),\n min Int64,\n max Int64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30582025-04-30T06:29:55.656ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt64),\n counts Array(UInt64),\n min UInt64,\n max UInt64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30592025-04-30T06:29:55.661ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float32),\n counts Array(UInt64),\n min Float32,\n max Float32,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30602025-04-30T06:29:55.678ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float64),\n counts Array(UInt64),\n min Float64,\n max Float64,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30612025-04-30T06:29:55.685ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\n/* The field tables store named dimensions of each timeseries.\n *\n * As with the measurement tables, there is one field table for each field data\n * type. Fields are deduplicated by using the "replacing merge tree", though\n * this behavior **must not** be relied upon for query correctness.\n *\n * The index for the fields differs from the measurements, however. Rows are\n * sorted by timeseries name, then field name, field value, and finally\n * timeseries key. This reflects the most common pattern for looking them up:\n * by field name and possibly value, within a timeseries. The resulting keys are\n * usually then used to look up measurements.\n *\n * NOTE: We may want to consider a secondary index on these tables, sorting by\n * timeseries name and then key, since it would improve lookups where one\n * already has the key. Realistically though, these tables are quite small and\n * so performance benefits will be low in absolute terms.\n *\n * TTL: We use a materialized column to expire old field table records. This\n * column is generated automatically by the database whenever a new row is\n * inserted. It cannot be inserted directly, nor is it returned in a `SELECT *`\n * query. Since these tables are `ReplacingMergeTree`s, that means the last\n * record will remain during a deduplication, which will have the last\n * timestamp. ClickHouse will then expire old data for us, similar to the\n * measurement tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.fields_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Bool,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30622025-04-30T06:29:55.688ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30632025-04-30T06:29:55.691ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30642025-04-30T06:29:55.696ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30652025-04-30T06:29:55.700ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30662025-04-30T06:29:55.704ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30672025-04-30T06:29:55.708ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30682025-04-30T06:29:55.714ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30692025-04-30T06:29:55.714ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
30702025-04-30T06:29:55.715ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
30712025-04-30T06:29:55.715ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.885162954s iteration = 3 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
30722025-04-30T06:29:55.718ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30732025-04-30T06:29:55.721ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_ipaddr\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value IPv6,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30742025-04-30T06:29:55.725ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value String,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30752025-04-30T06:29:55.733ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_uuid\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UUID,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30762025-04-30T06:29:55.736ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = \n\n/* The timeseries schema table stores the extracted schema for the samples\n * oximeter collects.\n */\nCREATE TABLE IF NOT EXISTS oximeter.timeseries_schema\n(\n timeseries_name String,\n fields Nested(\n name String,\n type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'IpAddr' = 3,\n 'String' = 4,\n 'Uuid' = 6,\n 'I8' = 7,\n 'U8' = 8,\n 'I16' = 9,\n 'U16' = 10,\n 'I32' = 11,\n 'U32' = 12,\n 'U64' = 13\n ),\n source Enum(\n 'Target' = 1,\n 'Metric' = 2\n )\n ),\n datum_type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'F64' = 3,\n 'String' = 4,\n 'Bytes' = 5,\n 'CumulativeI64' = 6,\n 'CumulativeF64' = 7,\n 'HistogramI64' = 8,\n 'HistogramF64' = 9,\n 'I8' = 10,\n 'U8' = 11,\n 'I16' = 12,\n 'U16' = 13,\n 'I32' = 14,\n 'U32' = 15,\n 'U64' = 16,\n 'F32' = 17,\n 'CumulativeU64' = 18,\n 'CumulativeF32' = 19,\n 'HistogramI8' = 20,\n 'HistogramU8' = 21,\n 'HistogramI16' = 22,\n 'HistogramU16' = 23,\n 'HistogramI32' = 24,\n 'HistogramU32' = 25,\n 'HistogramU64' = 26,\n 'HistogramF32' = 27\n ),\n created DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, fields.name)
30772025-04-30T06:29:55.739ZINFOtest_image_promotion (clickhouse-client): inserting current version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:825 id = a716977f-f822-467e-b942-62720757c880 version = 14
30782025-04-30T06:29:55.739ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = INSERT INTO oximeter.version (*) VALUES (14, now());
30792025-04-30T06:29:55.745ZDEBGtest_image_promotion (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f7ac2438-a3d7-426b-a357-ba0c9e62752a
30802025-04-30T06:29:55.746ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:34376 method = GET path = /info versions = all
30812025-04-30T06:29:55.746ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:34376 method = GET path = /producers versions = all
30822025-04-30T06:29:55.746ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:34376 method = DELETE path = /producers/{producer_id} versions = all
30832025-04-30T06:29:55.746ZDEBGtest_image_promotion (dropshot): registered endpoint local_addr = [::1]:34376 method = GET path = /producers/{producer_id} versions = all
30842025-04-30T06:29:55.747ZINFOtest_image_promotion (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [::1]:34376
30852025-04-30T06:29:55.748ZDEBGtest_image_promotion (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:34376
30862025-04-30T06:29:55.749ZDEBGtest_image_promotion (oximeter): registered USDT probes
30872025-04-30T06:29:55.749ZDEBGtest_image_promotion (oximeter): contacting nexus
30882025-04-30T06:29:56.329ZDEBGtest_image_promotion (oximeter): client request body = Some(Body) method = POST uri = http://127.0.0.1:45630/metrics/collectors
30892025-04-30T06:29:56.330ZDEBGtest_image_promotion (producer-registration-task): will register with Nexus at 127.0.0.1:45630
30902025-04-30T06:29:56.412ZDEBGtest_image_promotion (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:45630/metrics/producers
30912025-04-30T06:29:56.414ZINFOtest_image_promotion (dropshot_internal): accepted connection local_addr = 127.0.0.1:45630 remote_addr = 127.0.0.1:34622
30922025-04-30T06:29:56.414ZINFOtest_image_promotion (dropshot_internal): accepted connection local_addr = 127.0.0.1:45630 remote_addr = 127.0.0.1:47690
30932025-04-30T06:29:56.418ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:47690 req_id = 6c3fdcf7-90f4-4cd2-a5ab-2c691960efa0 uri = /metrics/producers
30942025-04-30T06:29:56.421ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:47690 req_id = 6c3fdcf7-90f4-4cd2-a5ab-2c691960efa0 resource = Database uri = /metrics/producers
30952025-04-30T06:29:56.422ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:47690 req_id = 6c3fdcf7-90f4-4cd2-a5ab-2c691960efa0 roles = RoleSet { roles: {} } uri = /metrics/producers
30962025-04-30T06:29:56.424ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:47690 req_id = 6c3fdcf7-90f4-4cd2-a5ab-2c691960efa0 resource = Database result = Ok(()) uri = /metrics/producers
30972025-04-30T06:29:56.425ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:34622 req_id = 66a6bbf4-240e-4a71-a6c7-155f24fbd540 uri = /metrics/collectors
30982025-04-30T06:29:56.426ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:34622 req_id = 66a6bbf4-240e-4a71-a6c7-155f24fbd540 resource = Database uri = /metrics/collectors
30992025-04-30T06:29:56.426ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:34622 req_id = 66a6bbf4-240e-4a71-a6c7-155f24fbd540 roles = RoleSet { roles: {} } uri = /metrics/collectors
31002025-04-30T06:29:56.427ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:34622 req_id = 66a6bbf4-240e-4a71-a6c7-155f24fbd540 resource = Database result = Ok(()) uri = /metrics/collectors
31012025-04-30T06:29:56.459ZINFOtest_image_promotion (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 41862 local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:47690 req_id = 6c3fdcf7-90f4-4cd2-a5ab-2c691960efa0 response_code = 503 uri = /metrics/producers
31022025-04-30T06:29:56.461ZDEBGtest_image_promotion (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "6c3fdcf7-90f4-4cd2-a5ab-2c691960efa0", "content-length": "133", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
31032025-04-30T06:29:56.463ZWARNtest_image_promotion (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 1.730173958s error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"6c3fdcf7-90f4-4cd2-a5ab-2c691960efa0\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:29:56 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"6c3fdcf7-90f4-4cd2-a5ab-2c691960efa0\\" }" file = oximeter/producer/src/lib.rs:371
31042025-04-30T06:29:56.464ZINFOtest_image_promotion (nexus): registered new oximeter metric collection server address = [::1]:34376 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31052025-04-30T06:29:56.464ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 38655 local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:34622 req_id = 66a6bbf4-240e-4a71-a6c7-155f24fbd540 response_code = 204 uri = /metrics/collectors
31062025-04-30T06:29:56.481ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors", status: 204, headers: {"x-request-id": "66a6bbf4-240e-4a71-a6c7-155f24fbd540", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
31072025-04-30T06:29:56.482ZINFOtest_image_promotion (oximeter): oximeter registered with nexus file = oximeter/collector/src/lib.rs:392 id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31082025-04-30T06:29:56.482ZDEBGtest_image_promotion (init_with_steps): Running step start_producer_server
31092025-04-30T06:29:56.482ZDEBGtest_image_promotion: Starting test metric Producer Server
31102025-04-30T06:29:56.491ZDEBGtest_image_promotion (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:38510 method = GET path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31112025-04-30T06:29:56.491ZDEBGtest_image_promotion (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:38510 method = POST path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31122025-04-30T06:29:56.491ZDEBGtest_image_promotion (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:38510 method = DELETE path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31132025-04-30T06:29:56.491ZDEBGtest_image_promotion (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:38510 method = GET path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31142025-04-30T06:29:56.491ZDEBGtest_image_promotion (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:38510 method = GET path = /crucible/0/regions/{id}/snapshots server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31152025-04-30T06:29:56.491ZDEBGtest_image_promotion (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:38510 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31162025-04-30T06:29:56.491ZDEBGtest_image_promotion (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:38510 method = GET path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31172025-04-30T06:29:56.491ZDEBGtest_image_promotion (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:38510 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31182025-04-30T06:29:56.491ZDEBGtest_image_promotion (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:38510 method = POST path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31192025-04-30T06:29:56.491ZINFOtest_image_promotion (Simulated CrucibleAgent Dropshot Server): listening kind = storage local_addr = [::1]:38510 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31202025-04-30T06:29:56.492ZDEBGtest_image_promotion (Simulated CrucibleAgent Dropshot Server): successfully registered DTrace USDT probes kind = storage local_addr = [::1]:38510 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31212025-04-30T06:29:56.492ZINFOtest_image_promotion (SledAgent): Created Simulated Crucible Server address = [::1]:38510 kind = storage server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31222025-04-30T06:29:56.492ZINFOtest_image_promotion (nexus): inserting test physical disk model = test-model physical_disk_id = e99bdd16-229a-4475-9033-497baeb2b69c serial = totally-unique-serial: e99bdd16-229a-4475-9033-497baeb2b69c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31232025-04-30T06:29:56.492ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31242025-04-30T06:29:56.492ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
31252025-04-30T06:29:56.493ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31262025-04-30T06:29:56.504ZINFOtest_image_promotion (oximeter-agent): starting refresh list task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:559
31272025-04-30T06:29:56.602ZINFOtest_image_promotion (dropshot_internal): accepted connection local_addr = 127.0.0.1:45630 remote_addr = 127.0.0.1:37297
31282025-04-30T06:29:56.602ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
31292025-04-30T06:29:56.605ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31302025-04-30T06:29:56.606ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 9684929f-092b-427c-a920-8d4249e302a6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31312025-04-30T06:29:56.606ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:37297 req_id = 8cb6e952-b6e2-4440-8d27-52bd711502cc uri = /metrics/producers
31322025-04-30T06:29:56.607ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 9684929f-092b-427c-a920-8d4249e302a6 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31332025-04-30T06:29:56.607ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 9684929f-092b-427c-a920-8d4249e302a6 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31342025-04-30T06:29:56.609ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 9684929f-092b-427c-a920-8d4249e302a6 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31352025-04-30T06:29:56.609ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:37297 req_id = 8cb6e952-b6e2-4440-8d27-52bd711502cc resource = Database uri = /metrics/producers
31362025-04-30T06:29:56.609ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:37297 req_id = 8cb6e952-b6e2-4440-8d27-52bd711502cc roles = RoleSet { roles: {} } uri = /metrics/producers
31372025-04-30T06:29:56.609ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:37297 req_id = 8cb6e952-b6e2-4440-8d27-52bd711502cc resource = Database result = Ok(()) uri = /metrics/producers
31382025-04-30T06:29:56.610ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31392025-04-30T06:29:56.610ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
31402025-04-30T06:29:56.614ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31412025-04-30T06:29:56.622ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 16341 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 9684929f-092b-427c-a920-8d4249e302a6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31422025-04-30T06:29:56.633ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "9684929f-092b-427c-a920-8d4249e302a6", "content-length": "29", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
31432025-04-30T06:29:56.639ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 0 n_pruned_tasks = 0
31442025-04-30T06:29:56.640ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31452025-04-30T06:29:56.640ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31462025-04-30T06:29:56.642ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31472025-04-30T06:29:56.642ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
31482025-04-30T06:29:56.643ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31492025-04-30T06:29:56.655ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31502025-04-30T06:29:56.662ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31512025-04-30T06:29:56.679ZINFOtest_image_promotion (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
31522025-04-30T06:29:56.680ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 73664 local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:37297 req_id = 8cb6e952-b6e2-4440-8d27-52bd711502cc response_code = 201 uri = /metrics/producers
31532025-04-30T06:29:56.750ZTRACtest_image_promotion (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
31542025-04-30T06:29:56.750ZTRACtest_image_promotion (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
31552025-04-30T06:29:56.837ZINFOtest_image_promotion (nexus): upserting zpool physical_disk_id = e99bdd16-229a-4475-9033-497baeb2b69c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 8edd0e1e-0d43-4301-83de-5dddb52e439b
31562025-04-30T06:29:56.837ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31572025-04-30T06:29:56.837ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
31582025-04-30T06:29:56.839ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31592025-04-30T06:29:56.844ZTRACtest_image_promotion (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31602025-04-30T06:29:56.844ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31612025-04-30T06:29:56.846ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31622025-04-30T06:29:56.846ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
31632025-04-30T06:29:56.847ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31642025-04-30T06:29:56.875ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31652025-04-30T06:29:56.882ZDEBGtest_image_promotion (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31662025-04-30T06:29:56.892ZINFOtest_image_promotion (nexus): upserting Crucible dataset dataset_id = b5621d4a-df89-43fe-a5f1-4858353cda81 zpool_id = 8edd0e1e-0d43-4301-83de-5dddb52e439b
31672025-04-30T06:29:56.906ZINFOtest_image_promotion (nexus): inserting test physical disk model = test-model physical_disk_id = e99bdd16-229a-4475-9033-497baeb2b69c serial = totally-unique-serial: e99bdd16-229a-4475-9033-497baeb2b69c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31682025-04-30T06:29:56.907ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31692025-04-30T06:29:56.907ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
31702025-04-30T06:29:56.908ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31712025-04-30T06:29:56.913ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: e99bdd16-229a-4475-9033-497baeb2b69c (physical_disk), lookup_type: ById(e99bdd16-229a-4475-9033-497baeb2b69c) }
31722025-04-30T06:29:56.914ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31732025-04-30T06:29:56.916ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31742025-04-30T06:29:56.916ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
31752025-04-30T06:29:56.922ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31762025-04-30T06:29:56.967ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31772025-04-30T06:29:56.975ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: e99bdd16-229a-4475-9033-497baeb2b69c (physical_disk), lookup_type: ById(e99bdd16-229a-4475-9033-497baeb2b69c) } result = Ok(())
31782025-04-30T06:29:56.975ZINFOtest_image_promotion (nexus): upserting zpool physical_disk_id = e99bdd16-229a-4475-9033-497baeb2b69c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 8edd0e1e-0d43-4301-83de-5dddb52e439b
31792025-04-30T06:29:56.975ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31802025-04-30T06:29:56.975ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
31812025-04-30T06:29:56.976ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31822025-04-30T06:29:56.978ZTRACtest_image_promotion (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31832025-04-30T06:29:56.978ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31842025-04-30T06:29:56.978ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31852025-04-30T06:29:56.978ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
31862025-04-30T06:29:56.979ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31872025-04-30T06:29:56.987ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31882025-04-30T06:29:56.996ZDEBGtest_image_promotion (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31892025-04-30T06:29:57.005ZTRACtest_image_promotion (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31902025-04-30T06:29:57.006ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31912025-04-30T06:29:57.006ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31922025-04-30T06:29:57.006ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
31932025-04-30T06:29:57.007ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31942025-04-30T06:29:57.017ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31952025-04-30T06:29:57.050ZDEBGtest_image_promotion (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31962025-04-30T06:29:57.050ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31972025-04-30T06:29:57.051ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
31982025-04-30T06:29:57.051ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31992025-04-30T06:29:57.062ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
32002025-04-30T06:29:57.062ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32012025-04-30T06:29:57.062ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32022025-04-30T06:29:57.063ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
32032025-04-30T06:29:57.063ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32042025-04-30T06:29:57.064ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 4 reason = Signaled
32052025-04-30T06:29:57.064ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
32062025-04-30T06:29:57.064ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32072025-04-30T06:29:57.064ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32082025-04-30T06:29:57.064ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32092025-04-30T06:29:57.064ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32102025-04-30T06:29:57.072ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32112025-04-30T06:29:57.079ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
32122025-04-30T06:29:57.079ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32132025-04-30T06:29:57.079ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
32142025-04-30T06:29:57.079ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32152025-04-30T06:29:57.080ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32162025-04-30T06:29:57.090ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
32172025-04-30T06:29:57.090ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32182025-04-30T06:29:57.090ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32192025-04-30T06:29:57.090ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32202025-04-30T06:29:57.096ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32212025-04-30T06:29:57.096ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
32222025-04-30T06:29:57.097ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32232025-04-30T06:29:57.097ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
32242025-04-30T06:29:57.097ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
32252025-04-30T06:29:57.101ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4d ab 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 M............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54038 req_id = 26d5de3e-d446-4087-977b-0faeaa0ddf2d
32262025-04-30T06:29:57.101ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19883,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54038 req_id = 26d5de3e-d446-4087-977b-0faeaa0ddf2d
32272025-04-30T06:29:57.101ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
32282025-04-30T06:29:57.101ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32292025-04-30T06:29:57.101ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54038 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 26d5de3e-d446-4087-977b-0faeaa0ddf2d
32302025-04-30T06:29:57.101ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3461, tv_nsec: 898800131 } })
32312025-04-30T06:29:57.101ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
32322025-04-30T06:29:57.101ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e5 ff 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65025 req_id = 922a9109-818b-48a4-95e7-fbacdbb214e2
32332025-04-30T06:29:57.101ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58879,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65025 req_id = 922a9109-818b-48a4-95e7-fbacdbb214e2
32342025-04-30T06:29:57.102ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32352025-04-30T06:29:57.102ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65025 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 922a9109-818b-48a4-95e7-fbacdbb214e2
32362025-04-30T06:29:57.150ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
32372025-04-30T06:29:57.152ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5f 88 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c _............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57546 req_id = f3e2ddce-e82c-4707-bdee-7ff2be33f914
32382025-04-30T06:29:57.153ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24456,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57546 req_id = f3e2ddce-e82c-4707-bdee-7ff2be33f914
32392025-04-30T06:29:57.154ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
32402025-04-30T06:29:57.154ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24456, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57546 req_id = f3e2ddce-e82c-4707-bdee-7ff2be33f914
32412025-04-30T06:29:57.154ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
32422025-04-30T06:29:57.154ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
32432025-04-30T06:29:57.154ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
32442025-04-30T06:29:57.156ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebd8391d-f75f-4249-93e4-99079abf92de", "content-length": "835", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32452025-04-30T06:29:57.156ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
32462025-04-30T06:29:57.159ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "242ec9ba-6afa-4a1f-ab8b-2bf9d4cb9882", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32472025-04-30T06:29:57.160ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
32482025-04-30T06:29:57.166ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22df2eee-345e-495f-8ea9-870ec4ca7441", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32492025-04-30T06:29:57.166ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
32502025-04-30T06:29:57.168ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1f3b8a6-acf4-4b60-ab00-5b84e0368fd4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32512025-04-30T06:29:57.168ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
32522025-04-30T06:29:57.171ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f0580a1-b955-414d-b0a2-bf872f777781", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32532025-04-30T06:29:57.171ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
32542025-04-30T06:29:57.174ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53e1d4e2-1c82-4371-afa9-59e8a4be37d2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32552025-04-30T06:29:57.174ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
32562025-04-30T06:29:57.189ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64a6eb86-6267-4bc9-9f16-2da8a2e4d2cf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32572025-04-30T06:29:57.190ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
32582025-04-30T06:29:57.196ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dc39a9c-86c4-4805-94e7-7c9e97e150e3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32592025-04-30T06:29:57.196ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
32602025-04-30T06:29:57.198ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe22a318-21ea-4057-bb85-197652fbb22d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32612025-04-30T06:29:57.199ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
32622025-04-30T06:29:57.200ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c39001f-2847-4d5c-9213-1505c3b00e0c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32632025-04-30T06:29:57.200ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
32642025-04-30T06:29:57.205ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92de97e9-dfc7-45ef-b39f-20601815203f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32652025-04-30T06:29:57.205ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
32662025-04-30T06:29:57.207ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0435a9c0-5ee0-4b6e-8cda-c0136ee05fd2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32672025-04-30T06:29:57.207ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
32682025-04-30T06:29:57.209ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a288f57-c7d8-454c-ab5d-a3823077aec1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32692025-04-30T06:29:57.209ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
32702025-04-30T06:29:57.220ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
32712025-04-30T06:29:57.221ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e663114c-eb45-4e13-a98b-0ea8160a3ea3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32722025-04-30T06:29:57.221ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
32732025-04-30T06:29:57.227ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c107dc7-983c-4626-abdd-ebcd751cfd30", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32742025-04-30T06:29:57.227ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
32752025-04-30T06:29:57.231ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96b97a79-ca83-4658-b0e1-c416bf370207", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32762025-04-30T06:29:57.231ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
32772025-04-30T06:29:57.235ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "752d2001-f301-47ac-9dcb-f9fa5dcc5c2b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32782025-04-30T06:29:57.235ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
32792025-04-30T06:29:57.238ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9eaac6e5-1b94-4076-b475-139585ac9911", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32802025-04-30T06:29:57.239ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
32812025-04-30T06:29:57.242ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d35cdee7-f942-4084-96e6-41fa55e77e31", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32822025-04-30T06:29:57.242ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
32832025-04-30T06:29:57.243ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd742763-3215-4aa7-8ac3-4bdd5874a52d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32842025-04-30T06:29:57.243ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
32852025-04-30T06:29:57.246ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "634bbfac-0af7-4713-8468-e17dd370c550", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32862025-04-30T06:29:57.247ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
32872025-04-30T06:29:57.248ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5026fea5-7714-4fad-a8e9-8a706abfaa69", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32882025-04-30T06:29:57.248ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
32892025-04-30T06:29:57.249ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26763647-d55e-4ec3-a0c2-3c1a28964bdf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32902025-04-30T06:29:57.249ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
32912025-04-30T06:29:57.250ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ffb151-d4a6-4309-95ee-29613ae86c0d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32922025-04-30T06:29:57.250ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
32932025-04-30T06:29:57.253ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ecd7475-10a1-42f1-bda4-150b53f320f9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32942025-04-30T06:29:57.254ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
32952025-04-30T06:29:57.256ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d7f6ed0-1ff5-4c3b-bb66-aced86cfdf99", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32962025-04-30T06:29:57.256ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
32972025-04-30T06:29:57.259ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f7d9d42-c978-4b89-961e-c8b720f2f254", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
32982025-04-30T06:29:57.259ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
32992025-04-30T06:29:57.261ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc5f9050-2625-4d34-8721-bd9cc2c6926f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33002025-04-30T06:29:57.261ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
33012025-04-30T06:29:57.264ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be0235ba-abb9-4a13-b9cd-85987d484172", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33022025-04-30T06:29:57.265ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
33032025-04-30T06:29:57.269ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9f6804c-211f-425e-b912-adbcde6d6393", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33042025-04-30T06:29:57.269ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
33052025-04-30T06:29:57.271ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33062025-04-30T06:29:57.271ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33072025-04-30T06:29:57.271ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33082025-04-30T06:29:57.271ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
33092025-04-30T06:29:57.271ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33102025-04-30T06:29:57.272ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d389e917-86c5-41a4-9553-c37418bc029b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33112025-04-30T06:29:57.272ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
33122025-04-30T06:29:57.273ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "006832cc-6130-457d-914e-1b5a6e9f47c8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33132025-04-30T06:29:57.274ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
33142025-04-30T06:29:57.275ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33152025-04-30T06:29:57.285ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33162025-04-30T06:29:57.285ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33172025-04-30T06:29:57.285ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
33182025-04-30T06:29:57.286ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33192025-04-30T06:29:57.289ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33202025-04-30T06:29:57.289ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
33212025-04-30T06:29:57.290ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33222025-04-30T06:29:57.290ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d0364c2-cde9-4f41-8679-7f6551c1f307", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33232025-04-30T06:29:57.291ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
33242025-04-30T06:29:57.297ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8c0815f-c116-4828-b1fe-5b209b44cf3c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33252025-04-30T06:29:57.298ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
33262025-04-30T06:29:57.299ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bb918ba-c5fb-49a8-8d0d-d4137a4e5616", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33272025-04-30T06:29:57.300ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
33282025-04-30T06:29:57.305ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ccb5b72-b1e1-4e78-b868-c06d7a7eccf6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33292025-04-30T06:29:57.305ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
33302025-04-30T06:29:57.307ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b21d6b1-7682-49dd-8c6b-c300eb958062", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33312025-04-30T06:29:57.307ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
33322025-04-30T06:29:57.309ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3838bd29-23fb-477d-bcf4-43735c815839", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33332025-04-30T06:29:57.309ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
33342025-04-30T06:29:57.312ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "229a4799-fa6d-49a0-a446-d2f64ecb77f9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33352025-04-30T06:29:57.312ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
33362025-04-30T06:29:57.321ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0da5910e-1ab0-4575-ac2c-27ed23cc5534", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33372025-04-30T06:29:57.321ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
33382025-04-30T06:29:57.330ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f67cd26-cd44-4b7c-933f-dc946319a37f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33392025-04-30T06:29:57.330ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
33402025-04-30T06:29:57.331ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86ec77f9-3fd8-4a5d-99c5-734424e0a9c9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33412025-04-30T06:29:57.331ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
33422025-04-30T06:29:57.339ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a49ede3c-5e05-4176-8717-aba5d0e0aa98", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33432025-04-30T06:29:57.339ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
33442025-04-30T06:29:57.341ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
33452025-04-30T06:29:57.346ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b99d8261-f7ec-4742-8b3b-7cff0c5a19c6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33462025-04-30T06:29:57.346ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
33472025-04-30T06:29:57.348ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bd7547f-2bb6-4a67-8e5a-d9a8cfb06f9a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:56 GMT"} })
33482025-04-30T06:29:57.348ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33492025-04-30T06:29:57.348ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33502025-04-30T06:29:57.348ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33512025-04-30T06:29:57.348ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33522025-04-30T06:29:57.351ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33532025-04-30T06:29:57.356ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33542025-04-30T06:29:57.367ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33552025-04-30T06:29:57.367ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33562025-04-30T06:29:57.367ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33572025-04-30T06:29:57.367ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33582025-04-30T06:29:57.367ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33592025-04-30T06:29:57.370ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33602025-04-30T06:29:57.376ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33612025-04-30T06:29:57.383ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33622025-04-30T06:29:57.383ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33632025-04-30T06:29:57.383ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33642025-04-30T06:29:57.383ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33652025-04-30T06:29:57.396ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33662025-04-30T06:29:57.396ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33672025-04-30T06:29:57.396ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33682025-04-30T06:29:57.396ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
33692025-04-30T06:29:57.396ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33702025-04-30T06:29:57.399ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33712025-04-30T06:29:57.416ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33722025-04-30T06:29:57.416ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33732025-04-30T06:29:57.416ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
33742025-04-30T06:29:57.417ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33752025-04-30T06:29:57.420ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33762025-04-30T06:29:57.420ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
33772025-04-30T06:29:57.423ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33782025-04-30T06:29:57.523ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
33792025-04-30T06:29:57.523ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
33802025-04-30T06:29:57.530ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:63189 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
33812025-04-30T06:29:57.530ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:63189 req_id = 246913e6-1993-4c15-934a-72a7362a1803 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33822025-04-30T06:29:57.532ZINFOtest_image_promotion (dropshot): request completed latency_us = 1323 local_addr = [::1]:60070 method = GET remote_addr = [::1]:63189 req_id = 246913e6-1993-4c15-934a-72a7362a1803 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33832025-04-30T06:29:57.532ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "246913e6-1993-4c15-934a-72a7362a1803", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
33842025-04-30T06:29:57.535ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33852025-04-30T06:29:57.535ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33862025-04-30T06:29:57.535ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
33872025-04-30T06:29:57.536ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
33882025-04-30T06:29:57.536ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33892025-04-30T06:29:57.536ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33902025-04-30T06:29:57.537ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33912025-04-30T06:29:57.537ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33922025-04-30T06:29:57.541ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33932025-04-30T06:29:57.546ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
33942025-04-30T06:29:57.546ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33952025-04-30T06:29:57.546ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33962025-04-30T06:29:57.547ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33972025-04-30T06:29:57.594ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
33982025-04-30T06:29:57.645ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33992025-04-30T06:29:57.645ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34002025-04-30T06:29:57.645ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34012025-04-30T06:29:57.645ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
34022025-04-30T06:29:57.646ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34032025-04-30T06:29:57.648ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34042025-04-30T06:29:57.653ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34052025-04-30T06:29:57.653ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34062025-04-30T06:29:57.653ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
34072025-04-30T06:29:57.653ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34082025-04-30T06:29:57.725ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
34092025-04-30T06:29:57.725ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
34102025-04-30T06:29:57.725ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 661.478773ms iteration = 4 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
34112025-04-30T06:29:57.725ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 5 reason = Signaled
34122025-04-30T06:29:57.725ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34132025-04-30T06:29:57.725ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34142025-04-30T06:29:57.725ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34152025-04-30T06:29:57.725ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34162025-04-30T06:29:57.726ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34172025-04-30T06:29:57.728ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34182025-04-30T06:29:57.732ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34192025-04-30T06:29:57.732ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34202025-04-30T06:29:57.732ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34212025-04-30T06:29:57.733ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34222025-04-30T06:29:57.733ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34232025-04-30T06:29:57.733ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
34242025-04-30T06:29:57.733ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34252025-04-30T06:29:57.736ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
34262025-04-30T06:29:57.736ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
34272025-04-30T06:29:57.737ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ad 48 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .H..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64656 req_id = 3724d203-df8f-4082-a337-f85bcd1f1d4b
34282025-04-30T06:29:57.737ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44360,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64656 req_id = 3724d203-df8f-4082-a337-f85bcd1f1d4b
34292025-04-30T06:29:57.737ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
34302025-04-30T06:29:57.737ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34312025-04-30T06:29:57.737ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64656 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3724d203-df8f-4082-a337-f85bcd1f1d4b
34322025-04-30T06:29:57.738ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3462, tv_nsec: 536908441 } })
34332025-04-30T06:29:57.738ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
34342025-04-30T06:29:57.738ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 96 03 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55866 req_id = 1c77e9d5-0b63-469a-9464-a95d47c8fc04
34352025-04-30T06:29:57.738ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38403,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55866 req_id = 1c77e9d5-0b63-469a-9464-a95d47c8fc04
34362025-04-30T06:29:57.738ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34372025-04-30T06:29:57.738ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55866 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1c77e9d5-0b63-469a-9464-a95d47c8fc04
34382025-04-30T06:29:57.797ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
34392025-04-30T06:29:57.797ZTRACtest_image_promotion (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
34402025-04-30T06:29:57.797ZTRACtest_image_promotion (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
34412025-04-30T06:29:57.798ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 52 ea 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c R............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59715 req_id = d5b3a868-f3d9-4f8e-a850-7becc7df1002
34422025-04-30T06:29:57.798ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21226,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59715 req_id = d5b3a868-f3d9-4f8e-a850-7becc7df1002
34432025-04-30T06:29:57.798ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
34442025-04-30T06:29:57.798ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21226, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59715 req_id = d5b3a868-f3d9-4f8e-a850-7becc7df1002
34452025-04-30T06:29:57.798ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
34462025-04-30T06:29:57.798ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
34472025-04-30T06:29:57.798ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
34482025-04-30T06:29:57.801ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "07722ae7-3ea5-47c7-bf0b-cd9d2be20380", "content-length": "835", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34492025-04-30T06:29:57.801ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
34502025-04-30T06:29:57.803ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3965bc3-cd00-46b0-95f6-1e424265a58d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34512025-04-30T06:29:57.803ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
34522025-04-30T06:29:57.806ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb72f88e-b2e3-4644-a5cc-3869269e137d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34532025-04-30T06:29:57.806ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
34542025-04-30T06:29:57.809ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "362bc444-65c6-4a88-b19d-9a78960e541d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34552025-04-30T06:29:57.810ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
34562025-04-30T06:29:57.818ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ef3b802-09bc-4073-bbf9-74fd4c55db38", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34572025-04-30T06:29:57.818ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
34582025-04-30T06:29:57.824ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "243e0159-eece-4458-b6a7-1303714a8c6b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34592025-04-30T06:29:57.824ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
34602025-04-30T06:29:57.835ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
34612025-04-30T06:29:57.835ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7f2de6c-9eb6-4887-b4c8-127fe49a3b08", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34622025-04-30T06:29:57.835ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
34632025-04-30T06:29:57.839ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e86b2982-d875-4029-a796-7e3ab02ea0b5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34642025-04-30T06:29:57.840ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
34652025-04-30T06:29:57.841ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6da3b6c-c679-44f2-a2ed-45b40cbf237f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34662025-04-30T06:29:57.841ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
34672025-04-30T06:29:57.842ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eac7b3da-4ae5-4654-96c7-545623d5554a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34682025-04-30T06:29:57.843ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
34692025-04-30T06:29:57.845ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f77476b-6def-4b55-abaa-b9b40ff1c2e8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34702025-04-30T06:29:57.845ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
34712025-04-30T06:29:57.848ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d2ccded-e521-4072-89c6-6ced2aab45f2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34722025-04-30T06:29:57.849ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
34732025-04-30T06:29:57.853ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48de4c45-7100-4639-ae4d-567171ca6b2d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34742025-04-30T06:29:57.853ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
34752025-04-30T06:29:57.858ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d62ff2f-fadf-40a5-bfdd-58b874157d72", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34762025-04-30T06:29:57.859ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
34772025-04-30T06:29:57.863ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8723777f-9d85-4ad5-a6eb-cbab2d6d3ff8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34782025-04-30T06:29:57.863ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
34792025-04-30T06:29:57.865ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38b84845-76d6-4f89-a03a-35d9d5233898", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34802025-04-30T06:29:57.866ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
34812025-04-30T06:29:57.869ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b68253c1-1fcd-4686-911e-1b548609d328", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34822025-04-30T06:29:57.870ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
34832025-04-30T06:29:57.874ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a90c018e-e20d-4283-9896-fa27cc9f9697", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34842025-04-30T06:29:57.874ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
34852025-04-30T06:29:57.877ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d634b6e-3f09-4abe-9949-324e9162f26a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34862025-04-30T06:29:57.877ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
34872025-04-30T06:29:57.878ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5857da16-ed87-4b7f-a356-19e2080844d2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34882025-04-30T06:29:57.878ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
34892025-04-30T06:29:57.879ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "623c7410-3994-463b-9840-76d2ce60b4e1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34902025-04-30T06:29:57.880ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
34912025-04-30T06:29:57.881ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7672df85-6f10-4720-ab76-03647d65910b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34922025-04-30T06:29:57.881ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
34932025-04-30T06:29:57.883ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b58d64ba-43d0-4d57-9e8d-a1fa0385b682", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34942025-04-30T06:29:57.883ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
34952025-04-30T06:29:57.885ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e1463dc-a51e-4373-969b-74cc191581ac", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
34962025-04-30T06:29:57.885ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
34972025-04-30T06:29:57.886ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
34982025-04-30T06:29:57.886ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34992025-04-30T06:29:57.886ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35002025-04-30T06:29:57.886ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
35012025-04-30T06:29:57.887ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35022025-04-30T06:29:57.890ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2a59390-3a11-4190-8435-166aee66fa1f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35032025-04-30T06:29:57.890ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
35042025-04-30T06:29:57.890ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35052025-04-30T06:29:57.897ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35062025-04-30T06:29:57.897ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35072025-04-30T06:29:57.897ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
35082025-04-30T06:29:57.898ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35092025-04-30T06:29:57.902ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35102025-04-30T06:29:57.902ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
35112025-04-30T06:29:57.903ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35122025-04-30T06:29:57.903ZDEBGtest_image_promotion (producer-registration-task): will register with Nexus at 127.0.0.1:45630
35132025-04-30T06:29:57.969ZDEBGtest_image_promotion (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:45630/metrics/producers
35142025-04-30T06:29:57.970ZINFOtest_image_promotion (dropshot_internal): accepted connection local_addr = 127.0.0.1:45630 remote_addr = 127.0.0.1:43038
35152025-04-30T06:29:57.970ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c94250a8-ae15-49b8-9dd8-32b50a1229b9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35162025-04-30T06:29:57.970ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
35172025-04-30T06:29:57.972ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:43038 req_id = e5d744a8-85d3-4947-a561-f45683e6d550 uri = /metrics/producers
35182025-04-30T06:29:57.973ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:43038 req_id = e5d744a8-85d3-4947-a561-f45683e6d550 resource = Database uri = /metrics/producers
35192025-04-30T06:29:57.973ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:43038 req_id = e5d744a8-85d3-4947-a561-f45683e6d550 roles = RoleSet { roles: {} } uri = /metrics/producers
35202025-04-30T06:29:57.974ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:43038 req_id = e5d744a8-85d3-4947-a561-f45683e6d550 resource = Database result = Ok(()) uri = /metrics/producers
35212025-04-30T06:29:57.984ZINFOtest_image_promotion (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
35222025-04-30T06:29:57.984ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 11795 local_addr = 127.0.0.1:45630 method = POST remote_addr = 127.0.0.1:43038 req_id = e5d744a8-85d3-4947-a561-f45683e6d550 response_code = 201 uri = /metrics/producers
35232025-04-30T06:29:57.989ZDEBGtest_image_promotion (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "e5d744a8-85d3-4947-a561-f45683e6d550", "content-length": "41", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35242025-04-30T06:29:57.991ZDEBGtest_image_promotion (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
35252025-04-30T06:29:58.033ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "250b854a-46ab-4ea3-aeb6-8bd7908069bd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35262025-04-30T06:29:58.033ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
35272025-04-30T06:29:58.051ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
35282025-04-30T06:29:58.053ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fddebe4a-d2b0-480f-b5e1-aadf6fac625a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35292025-04-30T06:29:58.054ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
35302025-04-30T06:29:58.057ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06791d3e-0810-4b0f-be18-7de8a3e44a60", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35312025-04-30T06:29:58.058ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
35322025-04-30T06:29:58.063ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bea2bcda-b44c-496b-993b-dbd2157ea21c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35332025-04-30T06:29:58.063ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
35342025-04-30T06:29:58.064ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c95a0e2-a969-404a-9929-46e56976600c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35352025-04-30T06:29:58.065ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
35362025-04-30T06:29:58.066ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af708666-7502-4364-873e-83ac277ded7a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35372025-04-30T06:29:58.066ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
35382025-04-30T06:29:58.067ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d72b73af-3c40-4140-bea2-6196b5575b5c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35392025-04-30T06:29:58.067ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
35402025-04-30T06:29:58.068ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39811d8d-4712-4369-a81b-9f7fcaafd1bd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35412025-04-30T06:29:58.068ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
35422025-04-30T06:29:58.069ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2635ea8c-0947-4e65-b428-dcca3dc20f13", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35432025-04-30T06:29:58.070ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
35442025-04-30T06:29:58.071ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "856efb53-5950-49a6-80b1-16c29e7ec4bd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35452025-04-30T06:29:58.071ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
35462025-04-30T06:29:58.073ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "843824e2-9797-4f47-8842-eeb7f9ac0a7f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35472025-04-30T06:29:58.074ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
35482025-04-30T06:29:58.076ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b77dc27-061a-4043-bc44-7d93c5ae372d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35492025-04-30T06:29:58.076ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
35502025-04-30T06:29:58.078ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07ebff10-df1d-476e-9adc-d395c9e5221b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35512025-04-30T06:29:58.078ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
35522025-04-30T06:29:58.080ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c26517e4-7da4-4620-894c-d7f691364a64", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35532025-04-30T06:29:58.080ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
35542025-04-30T06:29:58.082ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71540492-2f25-475d-9385-7006e562cc7d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35552025-04-30T06:29:58.082ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
35562025-04-30T06:29:58.083ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a99d51d6-8f9e-461e-8752-bb15f89bd846", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35572025-04-30T06:29:58.083ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
35582025-04-30T06:29:58.084ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb6d31e6-b541-430b-91c8-5b906863adef", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35592025-04-30T06:29:58.086ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
35602025-04-30T06:29:58.087ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f01b2f4-96a9-4eaa-8739-8e34301f1278", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35612025-04-30T06:29:58.087ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
35622025-04-30T06:29:58.088ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ed8398f-e343-490c-ae2d-f1dee3be711c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35632025-04-30T06:29:58.089ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35642025-04-30T06:29:58.089ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35652025-04-30T06:29:58.089ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35662025-04-30T06:29:58.089ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35672025-04-30T06:29:58.090ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35682025-04-30T06:29:58.100ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35692025-04-30T06:29:58.106ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35702025-04-30T06:29:58.106ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35712025-04-30T06:29:58.106ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35722025-04-30T06:29:58.106ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35732025-04-30T06:29:58.106ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35742025-04-30T06:29:58.107ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35752025-04-30T06:29:58.108ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35762025-04-30T06:29:58.109ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35772025-04-30T06:29:58.109ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35782025-04-30T06:29:58.109ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
35792025-04-30T06:29:58.109ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35802025-04-30T06:29:58.111ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35812025-04-30T06:29:58.117ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35822025-04-30T06:29:58.117ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35832025-04-30T06:29:58.117ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35842025-04-30T06:29:58.118ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35852025-04-30T06:29:58.124ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35862025-04-30T06:29:58.140ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35872025-04-30T06:29:58.140ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35882025-04-30T06:29:58.140ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
35892025-04-30T06:29:58.141ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35902025-04-30T06:29:58.201ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
35912025-04-30T06:29:58.201ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
35922025-04-30T06:29:58.202ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:36634 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
35932025-04-30T06:29:58.203ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:36634 req_id = cf3efef6-7c0c-454a-ae87-34fc60b144a0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35942025-04-30T06:29:58.203ZINFOtest_image_promotion (dropshot): request completed latency_us = 636 local_addr = [::1]:60070 method = GET remote_addr = [::1]:36634 req_id = cf3efef6-7c0c-454a-ae87-34fc60b144a0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35952025-04-30T06:29:58.204ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35962025-04-30T06:29:58.204ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
35972025-04-30T06:29:58.206ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35982025-04-30T06:29:58.207ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf3efef6-7c0c-454a-ae87-34fc60b144a0", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
35992025-04-30T06:29:58.209ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
36002025-04-30T06:29:58.209ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
36012025-04-30T06:29:58.209ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
36022025-04-30T06:29:58.212ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36032025-04-30T06:29:58.212ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36042025-04-30T06:29:58.212ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36052025-04-30T06:29:58.212ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36062025-04-30T06:29:58.212ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36072025-04-30T06:29:58.216ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36082025-04-30T06:29:58.224ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36092025-04-30T06:29:58.225ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36102025-04-30T06:29:58.225ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36112025-04-30T06:29:58.225ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36122025-04-30T06:29:58.263ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
36132025-04-30T06:29:58.265ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
36142025-04-30T06:29:58.265ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 540.291916ms iteration = 5 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
36152025-04-30T06:29:58.266ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 6 reason = Signaled
36162025-04-30T06:29:58.266ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36172025-04-30T06:29:58.266ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36182025-04-30T06:29:58.266ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36192025-04-30T06:29:58.266ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36202025-04-30T06:29:58.280ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36212025-04-30T06:29:58.283ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36222025-04-30T06:29:58.290ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36232025-04-30T06:29:58.290ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36242025-04-30T06:29:58.290ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36252025-04-30T06:29:58.292ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36262025-04-30T06:29:58.300ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
36272025-04-30T06:29:58.301ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
36282025-04-30T06:29:58.301ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
36292025-04-30T06:29:58.303ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4b 27 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 K'..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56514 req_id = e1982539-12c8-40aa-8d2a-a762ebc09c1a
36302025-04-30T06:29:58.303ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19239,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56514 req_id = e1982539-12c8-40aa-8d2a-a762ebc09c1a
36312025-04-30T06:29:58.303ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
36322025-04-30T06:29:58.303ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36332025-04-30T06:29:58.303ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56514 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e1982539-12c8-40aa-8d2a-a762ebc09c1a
36342025-04-30T06:29:58.304ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3463, tv_nsec: 103350138 } })
36352025-04-30T06:29:58.304ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
36362025-04-30T06:29:58.306ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fa 40 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .@..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50245 req_id = dc1daf68-8601-42ef-8312-01392d5bdf15
36372025-04-30T06:29:58.307ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64064,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50245 req_id = dc1daf68-8601-42ef-8312-01392d5bdf15
36382025-04-30T06:29:58.308ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36392025-04-30T06:29:58.308ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50245 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dc1daf68-8601-42ef-8312-01392d5bdf15
36402025-04-30T06:29:58.393ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
36412025-04-30T06:29:58.393ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36422025-04-30T06:29:58.393ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36432025-04-30T06:29:58.393ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36442025-04-30T06:29:58.393ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
36452025-04-30T06:29:58.394ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36462025-04-30T06:29:58.395ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2d a1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c -............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60298 req_id = 33aa50c5-bba2-46bc-b976-a84f35eb3baf
36472025-04-30T06:29:58.395ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11681,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60298 req_id = 33aa50c5-bba2-46bc-b976-a84f35eb3baf
36482025-04-30T06:29:58.395ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
36492025-04-30T06:29:58.395ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11681, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60298 req_id = 33aa50c5-bba2-46bc-b976-a84f35eb3baf
36502025-04-30T06:29:58.395ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
36512025-04-30T06:29:58.395ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
36522025-04-30T06:29:58.395ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
36532025-04-30T06:29:58.397ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36542025-04-30T06:29:58.402ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36552025-04-30T06:29:58.402ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36562025-04-30T06:29:58.402ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
36572025-04-30T06:29:58.402ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36582025-04-30T06:29:58.407ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bc4be0d-37b1-425d-a8ad-e11ac89c87ad", "content-length": "835", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36592025-04-30T06:29:58.407ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
36602025-04-30T06:29:58.409ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36612025-04-30T06:29:58.409ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
36622025-04-30T06:29:58.410ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36632025-04-30T06:29:58.411ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0001561-6262-44f4-9db3-c7c5cfa45075", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36642025-04-30T06:29:58.411ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
36652025-04-30T06:29:58.416ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc9f0fbd-84e5-4571-95a2-2d4e64844607", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36662025-04-30T06:29:58.416ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
36672025-04-30T06:29:58.418ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "467971f6-b4ae-45f8-bede-4c22b5d82214", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36682025-04-30T06:29:58.419ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
36692025-04-30T06:29:58.421ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb3a102a-5d34-4f1d-9999-90c10e3467c1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36702025-04-30T06:29:58.421ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
36712025-04-30T06:29:58.423ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c1a17b3-c5ee-4414-9573-c7c5db99c10e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36722025-04-30T06:29:58.423ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
36732025-04-30T06:29:58.426ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "134c9f32-7285-4bc4-84d2-227d2aef0fef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36742025-04-30T06:29:58.426ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
36752025-04-30T06:29:58.428ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d74d3dc-08bf-4676-b648-c0863c578f09", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36762025-04-30T06:29:58.428ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
36772025-04-30T06:29:58.430ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "413d5fca-6e8b-4234-9dd7-058bcc9ae8cf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36782025-04-30T06:29:58.430ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
36792025-04-30T06:29:58.431ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ef29e0a-0e46-418d-9960-7e114c584039", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36802025-04-30T06:29:58.432ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
36812025-04-30T06:29:58.435ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2673896a-2e57-4ae3-9b27-144ca0b9fff1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36822025-04-30T06:29:58.436ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
36832025-04-30T06:29:58.437ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a36e9d6-ff6e-429c-9efe-30afb4bcc5c9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36842025-04-30T06:29:58.437ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
36852025-04-30T06:29:58.439ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e71b9df-d673-49e6-a61c-d607260b549c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36862025-04-30T06:29:58.439ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
36872025-04-30T06:29:58.442ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dfd03b1-0b91-4831-b081-94c32abe697c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36882025-04-30T06:29:58.442ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
36892025-04-30T06:29:58.446ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22a4ef5a-df09-4262-8446-cca7f2f99d59", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36902025-04-30T06:29:58.446ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
36912025-04-30T06:29:58.447ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
36922025-04-30T06:29:58.450ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "623862ce-f3a0-446a-b67c-afe054b17789", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36932025-04-30T06:29:58.450ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
36942025-04-30T06:29:58.452ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cd45d91-1bf2-44dd-8072-4cd0cb309bac", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36952025-04-30T06:29:58.452ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
36962025-04-30T06:29:58.454ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c47ce535-005a-4544-87e2-0b873e0f9aad", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36972025-04-30T06:29:58.455ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
36982025-04-30T06:29:58.457ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff28d9cf-bf65-4b66-b1ef-287d5fdf3deb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
36992025-04-30T06:29:58.457ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
37002025-04-30T06:29:58.458ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "783df6ca-fa30-41ce-a67c-011912812cf5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37012025-04-30T06:29:58.458ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
37022025-04-30T06:29:58.460ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae88bbbb-4cad-4b41-a875-e3a93b113c5b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37032025-04-30T06:29:58.460ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
37042025-04-30T06:29:58.462ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4a40527-83f3-4e95-8245-cdf5dc57fae9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37052025-04-30T06:29:58.462ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
37062025-04-30T06:29:58.463ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc4e06cc-cebe-43ce-89c5-0c9cbd227459", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37072025-04-30T06:29:58.463ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
37082025-04-30T06:29:58.465ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f09ced1b-47ad-4798-8375-c99f8ee4c5b6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37092025-04-30T06:29:58.465ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
37102025-04-30T06:29:58.469ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f14df4c3-5f00-42fd-bc88-270bdd995ae8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37112025-04-30T06:29:58.470ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
37122025-04-30T06:29:58.475ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b0beaa5-b5a3-4e4b-afbb-bb2a36a9227e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37132025-04-30T06:29:58.476ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
37142025-04-30T06:29:58.478ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4502df14-4373-417c-8e3d-52d32b10b929", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37152025-04-30T06:29:58.479ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
37162025-04-30T06:29:58.493ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74036b72-ff95-4d5b-90d8-6c7366f5e6ea", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37172025-04-30T06:29:58.493ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
37182025-04-30T06:29:58.495ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76df3e8c-e899-48e7-b921-9c498d8a73e2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37192025-04-30T06:29:58.495ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
37202025-04-30T06:29:58.497ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37212025-04-30T06:29:58.498ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37222025-04-30T06:29:58.498ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37232025-04-30T06:29:58.498ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
37242025-04-30T06:29:58.499ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37252025-04-30T06:29:58.500ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3989597d-5c98-49ad-8981-cfdddf806978", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37262025-04-30T06:29:58.500ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
37272025-04-30T06:29:58.502ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de611b1d-c28f-43e5-a7d8-fac7dbce0f14", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37282025-04-30T06:29:58.502ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
37292025-04-30T06:29:58.503ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37302025-04-30T06:29:58.510ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37312025-04-30T06:29:58.510ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37322025-04-30T06:29:58.510ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
37332025-04-30T06:29:58.511ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37342025-04-30T06:29:58.512ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
37352025-04-30T06:29:58.513ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37362025-04-30T06:29:58.513ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 7aa6bebf-50a5-48ed-97b2-f4df4b939cb7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37372025-04-30T06:29:58.513ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 7aa6bebf-50a5-48ed-97b2-f4df4b939cb7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37382025-04-30T06:29:58.513ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 7aa6bebf-50a5-48ed-97b2-f4df4b939cb7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37392025-04-30T06:29:58.514ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 7aa6bebf-50a5-48ed-97b2-f4df4b939cb7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37402025-04-30T06:29:58.516ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "975b065e-88c0-4a36-a82d-70c49be81290", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37412025-04-30T06:29:58.516ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
37422025-04-30T06:29:58.517ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37432025-04-30T06:29:58.517ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
37442025-04-30T06:29:58.518ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37452025-04-30T06:29:58.520ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 6738 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 7aa6bebf-50a5-48ed-97b2-f4df4b939cb7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37462025-04-30T06:29:58.522ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "7aa6bebf-50a5-48ed-97b2-f4df4b939cb7", "content-length": "405", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37472025-04-30T06:29:58.522ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
37482025-04-30T06:29:58.522ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90a8520b-6bb9-46ef-b5f1-76333bb7fd96", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37492025-04-30T06:29:58.522ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
37502025-04-30T06:29:58.524ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 0ec2f91f-4f32-4ced-aa19-f3ffd89a3f9e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
37512025-04-30T06:29:58.525ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 0ec2f91f-4f32-4ced-aa19-f3ffd89a3f9e resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
37522025-04-30T06:29:58.525ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 0ec2f91f-4f32-4ced-aa19-f3ffd89a3f9e roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
37532025-04-30T06:29:58.525ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 0ec2f91f-4f32-4ced-aa19-f3ffd89a3f9e resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
37542025-04-30T06:29:58.529ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cc429df-0fce-44f1-ad4e-08652afc1b1c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37552025-04-30T06:29:58.529ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
37562025-04-30T06:29:58.530ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 6870 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 0ec2f91f-4f32-4ced-aa19-f3ffd89a3f9e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
37572025-04-30T06:29:58.530ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ec2f91f-4f32-4ced-aa19-f3ffd89a3f9e", "content-length": "29", "date": "Wed, 30 Apr 2025 06:29:57 GMT"} })
37582025-04-30T06:29:58.530ZDEBGtest_image_promotion (oximeter-agent): registered new metric producer address = [::1]:44947 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
37592025-04-30T06:29:58.641ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
37602025-04-30T06:29:58.643ZDEBGtest_image_promotion (oximeter-agent): registered new metric producer address = 127.0.0.1:60931 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
37612025-04-30T06:29:58.755ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
37622025-04-30T06:29:58.756ZTRACtest_image_promotion (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
37632025-04-30T06:29:58.756ZTRACtest_image_promotion (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
37642025-04-30T06:29:58.757ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a58f8b5-98f7-4e81-80cb-699c976afa66", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
37652025-04-30T06:29:58.757ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
37662025-04-30T06:29:58.757ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
37672025-04-30T06:29:58.760ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79d4d877-dcc3-4b9f-afaf-1f9847aa3b6c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
37682025-04-30T06:29:58.760ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
37692025-04-30T06:29:58.763ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5afc228-db41-4995-8dd8-1640d18d686c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
37702025-04-30T06:29:58.763ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
37712025-04-30T06:29:58.768ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e261ee2-dae7-4ca3-a115-cca19a7f443a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
37722025-04-30T06:29:58.768ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
37732025-04-30T06:29:58.781ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef49b210-ce7b-4cd3-8270-91c2d36dee23", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
37742025-04-30T06:29:58.781ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
37752025-04-30T06:29:58.797ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
37762025-04-30T06:29:58.801ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dabdcdb-f52c-4d81-b41e-1e304ea965d0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
37772025-04-30T06:29:58.801ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
37782025-04-30T06:29:58.804ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09667237-7737-4a0a-a912-e278f8ad93fc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
37792025-04-30T06:29:58.804ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
37802025-04-30T06:29:58.807ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd35a977-9630-4ac6-91b2-cc7b3768fbc5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
37812025-04-30T06:29:58.807ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
37822025-04-30T06:29:58.808ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dc155d9-858e-40f3-8a64-ee781bf191be", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
37832025-04-30T06:29:58.808ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
37842025-04-30T06:29:58.811ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6760425d-7bc8-4c32-83bb-e5563f5a771f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
37852025-04-30T06:29:58.811ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
37862025-04-30T06:29:58.814ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fb9653d-ccb8-4bab-90ba-e362287b1101", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
37872025-04-30T06:29:58.814ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37882025-04-30T06:29:58.814ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37892025-04-30T06:29:58.814ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37902025-04-30T06:29:58.814ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37912025-04-30T06:29:58.815ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37922025-04-30T06:29:58.818ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37932025-04-30T06:29:58.828ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
37942025-04-30T06:29:58.828ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37952025-04-30T06:29:58.828ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37962025-04-30T06:29:58.828ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37972025-04-30T06:29:58.828ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37982025-04-30T06:29:58.829ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37992025-04-30T06:29:58.833ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38002025-04-30T06:29:58.838ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
38012025-04-30T06:29:58.838ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38022025-04-30T06:29:58.838ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38032025-04-30T06:29:58.838ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38042025-04-30T06:29:58.928ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
38052025-04-30T06:29:58.928ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
38062025-04-30T06:29:58.929ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38072025-04-30T06:29:58.929ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38082025-04-30T06:29:58.929ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38092025-04-30T06:29:58.929ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
38102025-04-30T06:29:58.930ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38112025-04-30T06:29:58.930ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:49975 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
38122025-04-30T06:29:58.931ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:49975 req_id = 9d1263bb-6944-46f4-ba4f-f6fddffd129a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38132025-04-30T06:29:58.932ZINFOtest_image_promotion (dropshot): request completed latency_us = 493 local_addr = [::1]:60070 method = GET remote_addr = [::1]:49975 req_id = 9d1263bb-6944-46f4-ba4f-f6fddffd129a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38142025-04-30T06:29:58.932ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d1263bb-6944-46f4-ba4f-f6fddffd129a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
38152025-04-30T06:29:58.933ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38162025-04-30T06:29:58.934ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38172025-04-30T06:29:58.934ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
38182025-04-30T06:29:58.935ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38192025-04-30T06:29:58.936ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38202025-04-30T06:29:58.936ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38212025-04-30T06:29:58.936ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38222025-04-30T06:29:58.937ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38232025-04-30T06:29:58.937ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38242025-04-30T06:29:58.945ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38252025-04-30T06:29:58.945ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38262025-04-30T06:29:58.945ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
38272025-04-30T06:29:58.945ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38282025-04-30T06:29:58.947ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38292025-04-30T06:29:58.953ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38302025-04-30T06:29:58.956ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38312025-04-30T06:29:58.956ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38322025-04-30T06:29:58.957ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38332025-04-30T06:29:58.959ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38342025-04-30T06:29:58.959ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
38352025-04-30T06:29:58.960ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38362025-04-30T06:29:59.024ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
38372025-04-30T06:29:59.025ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
38382025-04-30T06:29:59.025ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 759.074306ms iteration = 6 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
38392025-04-30T06:29:59.025ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 7 reason = Signaled
38402025-04-30T06:29:59.025ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38412025-04-30T06:29:59.025ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38422025-04-30T06:29:59.025ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38432025-04-30T06:29:59.025ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38442025-04-30T06:29:59.026ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38452025-04-30T06:29:59.032ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38462025-04-30T06:29:59.037ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38472025-04-30T06:29:59.037ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38482025-04-30T06:29:59.037ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38492025-04-30T06:29:59.038ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38502025-04-30T06:29:59.038ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
38512025-04-30T06:29:59.042ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
38522025-04-30T06:29:59.042ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
38532025-04-30T06:29:59.043ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 00 bc 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54766 req_id = 778cab96-196a-4e0b-bbc8-59c5bfb5e4f0
38542025-04-30T06:29:59.043ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 188,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54766 req_id = 778cab96-196a-4e0b-bbc8-59c5bfb5e4f0
38552025-04-30T06:29:59.043ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
38562025-04-30T06:29:59.044ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38572025-04-30T06:29:59.044ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54766 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 778cab96-196a-4e0b-bbc8-59c5bfb5e4f0
38582025-04-30T06:29:59.044ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3463, tv_nsec: 843535028 } })
38592025-04-30T06:29:59.044ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
38602025-04-30T06:29:59.044ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 24 6e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e $n..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58623 req_id = d226867b-9b4d-497e-8e29-80b183cd3a02
38612025-04-30T06:29:59.045ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9326,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58623 req_id = d226867b-9b4d-497e-8e29-80b183cd3a02
38622025-04-30T06:29:59.045ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38632025-04-30T06:29:59.045ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58623 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d226867b-9b4d-497e-8e29-80b183cd3a02
38642025-04-30T06:29:59.109ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
38652025-04-30T06:29:59.110ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38662025-04-30T06:29:59.110ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38672025-04-30T06:29:59.110ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38682025-04-30T06:29:59.110ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
38692025-04-30T06:29:59.120ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38702025-04-30T06:29:59.120ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 21 84 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c !............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56637 req_id = b0351b74-18b2-4c65-bbb4-ee8599142de5
38712025-04-30T06:29:59.120ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8580,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56637 req_id = b0351b74-18b2-4c65-bbb4-ee8599142de5
38722025-04-30T06:29:59.121ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
38732025-04-30T06:29:59.121ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8580, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56637 req_id = b0351b74-18b2-4c65-bbb4-ee8599142de5
38742025-04-30T06:29:59.122ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
38752025-04-30T06:29:59.122ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
38762025-04-30T06:29:59.122ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
38772025-04-30T06:29:59.123ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38782025-04-30T06:29:59.131ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38792025-04-30T06:29:59.131ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38802025-04-30T06:29:59.131ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
38812025-04-30T06:29:59.131ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38822025-04-30T06:29:59.133ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "dca0fa9d-f414-4da4-9b2e-61b99b914e74", "content-length": "835", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
38832025-04-30T06:29:59.134ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
38842025-04-30T06:29:59.135ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38852025-04-30T06:29:59.135ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
38862025-04-30T06:29:59.136ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38872025-04-30T06:29:59.140ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37e2c22d-ec07-4f61-9e01-0a6639fc8951", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
38882025-04-30T06:29:59.141ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
38892025-04-30T06:29:59.146ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "efaa6c28-3842-418f-96a2-17660655f2d8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
38902025-04-30T06:29:59.146ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
38912025-04-30T06:29:59.150ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8363fab4-a83a-499c-bb07-3aee515e4e96", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
38922025-04-30T06:29:59.151ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
38932025-04-30T06:29:59.156ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "308e3acc-a815-4b4f-950d-3dd425ae10fa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
38942025-04-30T06:29:59.156ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
38952025-04-30T06:29:59.161ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37458e85-4022-49d6-9aee-aceda8e64dc4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
38962025-04-30T06:29:59.161ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
38972025-04-30T06:29:59.166ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
38982025-04-30T06:29:59.167ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96f12656-5944-45b0-9713-1d48d63222d7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
38992025-04-30T06:29:59.167ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
39002025-04-30T06:29:59.171ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a482fec8-7e5b-41cb-a62c-4ac46c01d6b5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39012025-04-30T06:29:59.171ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
39022025-04-30T06:29:59.172ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "817de1c5-868e-40f6-9252-eae7a0f13c92", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39032025-04-30T06:29:59.172ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
39042025-04-30T06:29:59.174ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d993b70-6b0a-4c3d-94bc-70e2622448b3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39052025-04-30T06:29:59.174ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
39062025-04-30T06:29:59.179ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0739bc74-90aa-4a05-b766-40b2e2c93ebf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39072025-04-30T06:29:59.180ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
39082025-04-30T06:29:59.185ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b45a01ed-fbd4-4349-8c05-d356ff23c355", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39092025-04-30T06:29:59.185ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
39102025-04-30T06:29:59.187ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbcc457b-303b-4764-baa9-2e18b11000b4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39112025-04-30T06:29:59.187ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
39122025-04-30T06:29:59.191ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00aff7f0-cd60-4ff7-bb38-7780e9a58072", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39132025-04-30T06:29:59.191ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
39142025-04-30T06:29:59.195ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8da8a1d6-d6a4-4fc8-a809-ca3cf7b6b2cb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39152025-04-30T06:29:59.195ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
39162025-04-30T06:29:59.199ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3af21850-8b54-4818-bb57-b6f893a4c77f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39172025-04-30T06:29:59.199ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
39182025-04-30T06:29:59.203ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21b58e12-378a-4c95-919e-f8b3d447d665", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39192025-04-30T06:29:59.203ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
39202025-04-30T06:29:59.207ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfdf4b65-905e-40cd-9e0a-17dd38c20f3b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39212025-04-30T06:29:59.207ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
39222025-04-30T06:29:59.237ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39232025-04-30T06:29:59.237ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39242025-04-30T06:29:59.238ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39252025-04-30T06:29:59.238ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
39262025-04-30T06:29:59.238ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39272025-04-30T06:29:59.240ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfeee646-7097-4a8f-b4dd-e23bf3961bcf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39282025-04-30T06:29:59.241ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
39292025-04-30T06:29:59.241ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39302025-04-30T06:29:59.248ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39312025-04-30T06:29:59.248ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39322025-04-30T06:29:59.249ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
39332025-04-30T06:29:59.249ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39342025-04-30T06:29:59.250ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "552842ea-dd8a-40a1-b87d-194ab5356230", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39352025-04-30T06:29:59.251ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
39362025-04-30T06:29:59.252ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39372025-04-30T06:29:59.252ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
39382025-04-30T06:29:59.253ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39392025-04-30T06:29:59.260ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9b2488b-027d-4676-8e35-141b2f445f4a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39402025-04-30T06:29:59.261ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
39412025-04-30T06:29:59.262ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ae7b11b-d036-4904-b966-4871b3213bc3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39422025-04-30T06:29:59.262ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
39432025-04-30T06:29:59.264ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3966c6ce-e9aa-44c7-aac5-5a92bad1708c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39442025-04-30T06:29:59.264ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
39452025-04-30T06:29:59.268ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d804a23-f6a3-429d-b91c-fe6ef46df663", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39462025-04-30T06:29:59.268ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
39472025-04-30T06:29:59.273ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b93300a-bb4c-46c6-a3b5-67e212f0b0ea", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39482025-04-30T06:29:59.274ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
39492025-04-30T06:29:59.280ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c29e4931-f418-47a0-b9ba-b85154e63c80", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39502025-04-30T06:29:59.280ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
39512025-04-30T06:29:59.287ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b8105ab-07ab-4933-b379-428b21680ea7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39522025-04-30T06:29:59.287ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
39532025-04-30T06:29:59.292ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fea7e92c-907c-4fb0-8a82-fa0574c9289e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39542025-04-30T06:29:59.292ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
39552025-04-30T06:29:59.298ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
39562025-04-30T06:29:59.298ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b4542d3-225d-4989-91d8-8e4653806ae9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39572025-04-30T06:29:59.298ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
39582025-04-30T06:29:59.304ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0180e5ce-26ad-4318-a3e7-c9b11516bd97", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39592025-04-30T06:29:59.305ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
39602025-04-30T06:29:59.306ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ff73783-d263-4a75-a27a-f22bc6432d8f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39612025-04-30T06:29:59.306ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
39622025-04-30T06:29:59.308ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "130ec461-198e-4523-b331-f556defca1f7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39632025-04-30T06:29:59.308ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
39642025-04-30T06:29:59.309ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ea75432-5ad6-4fb4-921b-887dc9fd8fdb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39652025-04-30T06:29:59.310ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
39662025-04-30T06:29:59.321ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0116528e-3289-4179-95c1-936aca4597d2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39672025-04-30T06:29:59.321ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
39682025-04-30T06:29:59.321ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec74788d-6851-4c8b-8e30-d4bbfbef617a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39692025-04-30T06:29:59.322ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
39702025-04-30T06:29:59.322ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce68fa93-f0b9-4195-a65a-904da373c061", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39712025-04-30T06:29:59.322ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
39722025-04-30T06:29:59.322ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3d131f6-05bf-4027-bd63-684f08181b3b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39732025-04-30T06:29:59.322ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
39742025-04-30T06:29:59.322ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2363bf11-fabf-442c-8031-ecc743176249", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39752025-04-30T06:29:59.322ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
39762025-04-30T06:29:59.324ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bd5e093-d5a5-4fce-9983-3922f2f543da", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39772025-04-30T06:29:59.324ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
39782025-04-30T06:29:59.328ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b661f834-494f-4ac6-8543-5a8ed0d5aba0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39792025-04-30T06:29:59.328ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
39802025-04-30T06:29:59.330ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74c13203-489d-494d-8d1a-5cf53bd8cf32", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39812025-04-30T06:29:59.330ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
39822025-04-30T06:29:59.331ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e233248-90e6-47a9-a164-645efdf81a0a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39832025-04-30T06:29:59.332ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
39842025-04-30T06:29:59.333ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40ca4df7-3c4d-4e8c-aae8-214ff94cb5d3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39852025-04-30T06:29:59.333ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
39862025-04-30T06:29:59.335ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08e013ac-6351-4ee0-a69f-19fe2d29d8a8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39872025-04-30T06:29:59.335ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
39882025-04-30T06:29:59.336ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a261c914-7655-45c0-a72d-5bcd0297ad32", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
39892025-04-30T06:29:59.336ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
39902025-04-30T06:29:59.336ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39912025-04-30T06:29:59.336ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39922025-04-30T06:29:59.336ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39932025-04-30T06:29:59.337ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39942025-04-30T06:29:59.342ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39952025-04-30T06:29:59.347ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
39962025-04-30T06:29:59.347ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
39972025-04-30T06:29:59.347ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39982025-04-30T06:29:59.347ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39992025-04-30T06:29:59.347ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40002025-04-30T06:29:59.347ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40012025-04-30T06:29:59.351ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40022025-04-30T06:29:59.351ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40032025-04-30T06:29:59.351ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40042025-04-30T06:29:59.351ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
40052025-04-30T06:29:59.351ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40062025-04-30T06:29:59.354ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40072025-04-30T06:29:59.358ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40082025-04-30T06:29:59.358ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40092025-04-30T06:29:59.358ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40102025-04-30T06:29:59.358ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40112025-04-30T06:29:59.368ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40122025-04-30T06:29:59.374ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40132025-04-30T06:29:59.374ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40142025-04-30T06:29:59.375ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
40152025-04-30T06:29:59.375ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40162025-04-30T06:29:59.452ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
40172025-04-30T06:29:59.452ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
40182025-04-30T06:29:59.453ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:49409 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
40192025-04-30T06:29:59.453ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:49409 req_id = fce0be26-b089-43a2-9489-c7b1118de97f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40202025-04-30T06:29:59.454ZINFOtest_image_promotion (dropshot): request completed latency_us = 606 local_addr = [::1]:60070 method = GET remote_addr = [::1]:49409 req_id = fce0be26-b089-43a2-9489-c7b1118de97f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40212025-04-30T06:29:59.454ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40222025-04-30T06:29:59.454ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
40232025-04-30T06:29:59.455ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40242025-04-30T06:29:59.456ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fce0be26-b089-43a2-9489-c7b1118de97f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
40252025-04-30T06:29:59.459ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40262025-04-30T06:29:59.459ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40272025-04-30T06:29:59.459ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
40282025-04-30T06:29:59.461ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40292025-04-30T06:29:59.461ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40302025-04-30T06:29:59.461ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40312025-04-30T06:29:59.461ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40322025-04-30T06:29:59.461ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40332025-04-30T06:29:59.465ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40342025-04-30T06:29:59.470ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40352025-04-30T06:29:59.470ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40362025-04-30T06:29:59.471ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40372025-04-30T06:29:59.471ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40382025-04-30T06:29:59.516ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
40392025-04-30T06:29:59.520ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
40402025-04-30T06:29:59.522ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
40412025-04-30T06:29:59.522ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 497.333324ms iteration = 7 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
40422025-04-30T06:29:59.522ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 8 reason = Signaled
40432025-04-30T06:29:59.522ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40442025-04-30T06:29:59.522ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40452025-04-30T06:29:59.522ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40462025-04-30T06:29:59.522ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40472025-04-30T06:29:59.523ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40482025-04-30T06:29:59.529ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40492025-04-30T06:29:59.535ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40502025-04-30T06:29:59.535ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40512025-04-30T06:29:59.535ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40522025-04-30T06:29:59.535ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40532025-04-30T06:29:59.539ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40542025-04-30T06:29:59.539ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40552025-04-30T06:29:59.539ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
40562025-04-30T06:29:59.539ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
40572025-04-30T06:29:59.541ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 21 51 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 !Q..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54365 req_id = d7815970-b9ee-43ec-a993-fe06be71bca7
40582025-04-30T06:29:59.541ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8529,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54365 req_id = d7815970-b9ee-43ec-a993-fe06be71bca7
40592025-04-30T06:29:59.543ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
40602025-04-30T06:29:59.543ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40612025-04-30T06:29:59.544ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54365 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d7815970-b9ee-43ec-a993-fe06be71bca7
40622025-04-30T06:29:59.544ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3464, tv_nsec: 342512077 } })
40632025-04-30T06:29:59.544ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
40642025-04-30T06:29:59.544ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40652025-04-30T06:29:59.545ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40662025-04-30T06:29:59.546ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40672025-04-30T06:29:59.546ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40682025-04-30T06:29:59.547ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 3a fd 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e :...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53339 req_id = 57f19f7b-cae2-497e-8e95-388913620fec
40692025-04-30T06:29:59.547ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15101,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53339 req_id = 57f19f7b-cae2-497e-8e95-388913620fec
40702025-04-30T06:29:59.547ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40712025-04-30T06:29:59.547ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53339 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 57f19f7b-cae2-497e-8e95-388913620fec
40722025-04-30T06:29:59.609ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
40732025-04-30T06:29:59.610ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40742025-04-30T06:29:59.610ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40752025-04-30T06:29:59.610ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40762025-04-30T06:29:59.617ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
40772025-04-30T06:29:59.619ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40782025-04-30T06:29:59.620ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 89 12 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63775 req_id = 2ed11b9c-240e-4087-9539-8416b5c7d513
40792025-04-30T06:29:59.620ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35090,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63775 req_id = 2ed11b9c-240e-4087-9539-8416b5c7d513
40802025-04-30T06:29:59.620ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
40812025-04-30T06:29:59.621ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35090, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63775 req_id = 2ed11b9c-240e-4087-9539-8416b5c7d513
40822025-04-30T06:29:59.621ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
40832025-04-30T06:29:59.621ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
40842025-04-30T06:29:59.621ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
40852025-04-30T06:29:59.622ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40862025-04-30T06:29:59.635ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40872025-04-30T06:29:59.635ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40882025-04-30T06:29:59.635ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
40892025-04-30T06:29:59.636ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40902025-04-30T06:29:59.644ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40912025-04-30T06:29:59.644ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
40922025-04-30T06:29:59.645ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40932025-04-30T06:29:59.645ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae62d99c-6368-4ed2-97f9-d7f85af4fc78", "content-length": "835", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
40942025-04-30T06:29:59.645ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
40952025-04-30T06:29:59.648ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7b0a8b1-46c4-43a9-920d-452081c3eb98", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
40962025-04-30T06:29:59.648ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
40972025-04-30T06:29:59.652ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bfb028a-f9e5-4edc-bc0c-bc9fdcddd697", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
40982025-04-30T06:29:59.652ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
40992025-04-30T06:29:59.659ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9347e818-45fc-4fa4-a464-d1b01291c4ed", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41002025-04-30T06:29:59.659ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
41012025-04-30T06:29:59.663ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aefd12f-00ce-47f1-a85f-69eeaef9ed51", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41022025-04-30T06:29:59.663ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
41032025-04-30T06:29:59.667ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0462fd87-041a-48e2-8e2e-dc000aa08744", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41042025-04-30T06:29:59.667ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
41052025-04-30T06:29:59.670ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf134ad5-4fc3-4adf-a8d8-fb7e52317386", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41062025-04-30T06:29:59.670ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
41072025-04-30T06:29:59.675ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
41082025-04-30T06:29:59.676ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea9a4ed2-d9e3-4b08-957e-2a7058adba1e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41092025-04-30T06:29:59.676ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
41102025-04-30T06:29:59.678ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cfef8ba-658f-4114-89c2-385d727b4180", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41112025-04-30T06:29:59.678ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
41122025-04-30T06:29:59.679ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbd03ea5-f214-4954-b231-3463009602f2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41132025-04-30T06:29:59.679ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
41142025-04-30T06:29:59.681ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c384854-6c75-4cc9-8826-b87570c8a3e2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41152025-04-30T06:29:59.681ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
41162025-04-30T06:29:59.682ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "998e0392-72d3-464a-b839-d3c5c425ce65", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41172025-04-30T06:29:59.682ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
41182025-04-30T06:29:59.683ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "050f0e8a-b632-4faf-893c-86dd6436f638", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41192025-04-30T06:29:59.684ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
41202025-04-30T06:29:59.686ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a4f2cf4-ede1-4f9f-8cff-e414a1f2d026", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41212025-04-30T06:29:59.686ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
41222025-04-30T06:29:59.687ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6888c51-a5b7-45b1-a5b0-4fd13c4f093b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41232025-04-30T06:29:59.688ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
41242025-04-30T06:29:59.690ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb6d18db-daeb-4586-bc95-36ffdb0fa235", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41252025-04-30T06:29:59.690ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
41262025-04-30T06:29:59.691ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef1f2bef-b0b5-4551-9885-e6b202cd501f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41272025-04-30T06:29:59.691ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
41282025-04-30T06:29:59.693ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "459162f4-a115-4470-8e51-ab037f01f3bd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41292025-04-30T06:29:59.694ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
41302025-04-30T06:29:59.695ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd081cf4-aa66-4a5c-b105-6cd0f861929f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41312025-04-30T06:29:59.695ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
41322025-04-30T06:29:59.696ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8277cfc-2db7-4bfc-a3fd-f755133060df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41332025-04-30T06:29:59.696ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
41342025-04-30T06:29:59.698ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93d22069-7368-4a5c-8ba9-dbe8e81a9b9e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41352025-04-30T06:29:59.698ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
41362025-04-30T06:29:59.699ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b50e3b4-34e4-49e1-9fcc-4ed0da1b4f21", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41372025-04-30T06:29:59.699ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
41382025-04-30T06:29:59.700ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f83027e-2d01-4ea7-afc6-2f2a371f5315", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41392025-04-30T06:29:59.700ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
41402025-04-30T06:29:59.701ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e694e59-0405-46bb-a4f4-c42a7e507a8e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41412025-04-30T06:29:59.701ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
41422025-04-30T06:29:59.703ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ea91d2d-2ae6-4b90-9749-96c95b5374fc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41432025-04-30T06:29:59.703ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
41442025-04-30T06:29:59.706ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc05f5b0-044b-47e0-9686-9fe7a1833b9b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41452025-04-30T06:29:59.706ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
41462025-04-30T06:29:59.708ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "651d7bee-3b45-4161-ad4d-3fd756b04e31", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41472025-04-30T06:29:59.708ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
41482025-04-30T06:29:59.711ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfcecffd-5bce-4fb8-bb83-41db1934a8aa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41492025-04-30T06:29:59.711ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
41502025-04-30T06:29:59.714ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c350dc9d-4615-4471-af1f-05111fb58fcb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41512025-04-30T06:29:59.714ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
41522025-04-30T06:29:59.716ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dcdd596-73d3-4b5d-bd0a-25094e5172c0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41532025-04-30T06:29:59.716ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
41542025-04-30T06:29:59.717ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95e1a50c-7211-4014-8c2c-ba6c87d44749", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41552025-04-30T06:29:59.717ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
41562025-04-30T06:29:59.718ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8374820-6c4c-40e5-ab45-a301c6e12f53", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41572025-04-30T06:29:59.719ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
41582025-04-30T06:29:59.719ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02eecd85-7f3d-4bfb-9582-962e25f3f566", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41592025-04-30T06:29:59.720ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
41602025-04-30T06:29:59.721ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94fdb2de-9b5a-4d2a-baea-d91fd6cb3dc3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41612025-04-30T06:29:59.721ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
41622025-04-30T06:29:59.722ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9350d0b-b2de-46be-89dd-f5634ee6e215", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41632025-04-30T06:29:59.722ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
41642025-04-30T06:29:59.725ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6f1c7e1-e901-4661-854b-cb190b88b849", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41652025-04-30T06:29:59.725ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
41662025-04-30T06:29:59.725ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41672025-04-30T06:29:59.726ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41682025-04-30T06:29:59.726ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41692025-04-30T06:29:59.726ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
41702025-04-30T06:29:59.726ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41712025-04-30T06:29:59.729ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41722025-04-30T06:29:59.734ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41732025-04-30T06:29:59.734ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41742025-04-30T06:29:59.734ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
41752025-04-30T06:29:59.735ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41762025-04-30T06:29:59.736ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "693a334a-4041-4da3-89cd-a836a80812e8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41772025-04-30T06:29:59.736ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
41782025-04-30T06:29:59.737ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41792025-04-30T06:29:59.737ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
41802025-04-30T06:29:59.737ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41812025-04-30T06:29:59.741ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "374237c2-24dc-46bb-a8bf-1c4ebc30e7ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41822025-04-30T06:29:59.741ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
41832025-04-30T06:29:59.743ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c5910ca-d5d3-4c54-9097-217abfa82b1b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41842025-04-30T06:29:59.743ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
41852025-04-30T06:29:59.745ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac8a363a-6d1d-434f-ac9b-e1ca7a13ca94", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41862025-04-30T06:29:59.745ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
41872025-04-30T06:29:59.747ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "170af743-0877-423c-8a54-a2c9662749b0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41882025-04-30T06:29:59.747ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
41892025-04-30T06:29:59.748ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3142571-dbde-4af1-a91d-deb2c19e5cde", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41902025-04-30T06:29:59.748ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
41912025-04-30T06:29:59.749ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0a8c30f-5cd9-4a58-b582-7b4349892174", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:58 GMT"} })
41922025-04-30T06:29:59.749ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
41932025-04-30T06:29:59.750ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
41942025-04-30T06:29:59.750ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
41952025-04-30T06:29:59.750ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
41962025-04-30T06:29:59.752ZDEBGtest_image_promotion (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
41972025-04-30T06:29:59.752ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = SELECT * FROM oximeter.timeseries_schema FORMAT Native;
41982025-04-30T06:29:59.756ZTRACtest_image_promotion (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
41992025-04-30T06:29:59.756ZDEBGtest_image_promotion (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 schema = TimeseriesSchema { timeseries_name: TimeseriesName("integration_target:integration_metric"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "metric_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "target_name", field_type: String, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:29:59.751999850Z } timeseries_name = integration_target:integration_metric
42002025-04-30T06:29:59.756ZDEBGtest_image_promotion (clickhouse-client): inserting 1 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
42012025-04-30T06:29:59.756ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 6 n_rows = 1 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
42022025-04-30T06:29:59.757ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "446f2735-78c7-4eed-8d17-5c84986b4457", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
42032025-04-30T06:29:59.758ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
42042025-04-30T06:29:59.761ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb935289-d517-4a37-ba22-800092587e1e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
42052025-04-30T06:29:59.761ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42062025-04-30T06:29:59.761ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42072025-04-30T06:29:59.761ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42082025-04-30T06:29:59.761ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42092025-04-30T06:29:59.762ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42102025-04-30T06:29:59.764ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
42112025-04-30T06:29:59.766ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42122025-04-30T06:29:59.770ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42132025-04-30T06:29:59.770ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42142025-04-30T06:29:59.770ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42152025-04-30T06:29:59.770ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42162025-04-30T06:29:59.771ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42172025-04-30T06:29:59.771ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42182025-04-30T06:29:59.772ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
42192025-04-30T06:29:59.773ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
42202025-04-30T06:29:59.776ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42212025-04-30T06:29:59.782ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42222025-04-30T06:29:59.782ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42232025-04-30T06:29:59.782ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42242025-04-30T06:29:59.783ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42252025-04-30T06:29:59.784ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
42262025-04-30T06:29:59.784ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42272025-04-30T06:29:59.854ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
42282025-04-30T06:29:59.854ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
42292025-04-30T06:29:59.856ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:56162 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
42302025-04-30T06:29:59.857ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
42312025-04-30T06:29:59.858ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
42322025-04-30T06:29:59.858ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:56162 req_id = deb61d46-03e8-43b4-b5e3-43e68d8354e4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42332025-04-30T06:29:59.859ZINFOtest_image_promotion (dropshot): request completed latency_us = 987 local_addr = [::1]:60070 method = GET remote_addr = [::1]:56162 req_id = deb61d46-03e8-43b4-b5e3-43e68d8354e4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42342025-04-30T06:29:59.860ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "deb61d46-03e8-43b4-b5e3-43e68d8354e4", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
42352025-04-30T06:29:59.862ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42362025-04-30T06:29:59.862ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42372025-04-30T06:29:59.862ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
42382025-04-30T06:29:59.864ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42392025-04-30T06:29:59.865ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42402025-04-30T06:29:59.865ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42412025-04-30T06:29:59.865ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42422025-04-30T06:29:59.866ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42432025-04-30T06:29:59.880ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42442025-04-30T06:29:59.889ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42452025-04-30T06:29:59.889ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42462025-04-30T06:29:59.889ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42472025-04-30T06:29:59.890ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42482025-04-30T06:29:59.910ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42492025-04-30T06:29:59.910ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42502025-04-30T06:29:59.910ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42512025-04-30T06:29:59.910ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
42522025-04-30T06:29:59.911ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42532025-04-30T06:29:59.926ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42542025-04-30T06:29:59.935ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42552025-04-30T06:29:59.935ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42562025-04-30T06:29:59.935ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
42572025-04-30T06:29:59.936ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42582025-04-30T06:30:00.016ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
42592025-04-30T06:30:00.017ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
42602025-04-30T06:30:00.017ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 495.336285ms iteration = 8 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
42612025-04-30T06:30:00.018ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 9 reason = Signaled
42622025-04-30T06:30:00.018ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42632025-04-30T06:30:00.018ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42642025-04-30T06:30:00.018ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42652025-04-30T06:30:00.018ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42662025-04-30T06:30:00.019ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42672025-04-30T06:30:00.022ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42682025-04-30T06:30:00.022ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
42692025-04-30T06:30:00.023ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42702025-04-30T06:30:00.025ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42712025-04-30T06:30:00.032ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42722025-04-30T06:30:00.033ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42732025-04-30T06:30:00.033ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42742025-04-30T06:30:00.033ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42752025-04-30T06:30:00.039ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
42762025-04-30T06:30:00.040ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
42772025-04-30T06:30:00.043ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fa c5 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58436 req_id = ce96424a-bcc6-45f5-9e1c-927a7761e17d
42782025-04-30T06:30:00.043ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64197,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58436 req_id = ce96424a-bcc6-45f5-9e1c-927a7761e17d
42792025-04-30T06:30:00.043ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
42802025-04-30T06:30:00.043ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42812025-04-30T06:30:00.043ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58436 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ce96424a-bcc6-45f5-9e1c-927a7761e17d
42822025-04-30T06:30:00.045ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3464, tv_nsec: 843920039 } })
42832025-04-30T06:30:00.045ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
42842025-04-30T06:30:00.045ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c6 bf 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56330 req_id = d1d561d8-dadc-4700-9da7-f4b0f649e74e
42852025-04-30T06:30:00.045ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50879,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56330 req_id = d1d561d8-dadc-4700-9da7-f4b0f649e74e
42862025-04-30T06:30:00.046ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42872025-04-30T06:30:00.046ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56330 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d1d561d8-dadc-4700-9da7-f4b0f649e74e
42882025-04-30T06:30:00.115ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
42892025-04-30T06:30:00.115ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 8d 08 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60355 req_id = c1738856-a611-4357-837c-849253062877
42902025-04-30T06:30:00.116ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36104,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60355 req_id = c1738856-a611-4357-837c-849253062877
42912025-04-30T06:30:00.116ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
42922025-04-30T06:30:00.116ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36104, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60355 req_id = c1738856-a611-4357-837c-849253062877
42932025-04-30T06:30:00.116ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
42942025-04-30T06:30:00.116ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
42952025-04-30T06:30:00.116ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
42962025-04-30T06:30:00.121ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "20843215-4ded-42ef-83d0-6a671302b976", "content-length": "835", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
42972025-04-30T06:30:00.121ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
42982025-04-30T06:30:00.124ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3918314-c195-4808-abf3-3cf450cbd516", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
42992025-04-30T06:30:00.124ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
43002025-04-30T06:30:00.135ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "994ae433-9c93-45b0-a748-d4b992b384c2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43012025-04-30T06:30:00.135ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
43022025-04-30T06:30:00.145ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
43032025-04-30T06:30:00.148ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "29a263cf-96f2-48af-95bb-56a9ac7e688f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43042025-04-30T06:30:00.148ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
43052025-04-30T06:30:00.151ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28bc3e32-cb5f-4011-977c-deea3f9f19e9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43062025-04-30T06:30:00.151ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
43072025-04-30T06:30:00.154ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9a8802d-121a-44e4-90dd-8b217751785b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43082025-04-30T06:30:00.154ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
43092025-04-30T06:30:00.158ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0865f613-4762-4e9a-b102-796708f75a45", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43102025-04-30T06:30:00.158ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
43112025-04-30T06:30:00.160ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7958a0b-f9f6-4f74-8d9d-990da503d2ae", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43122025-04-30T06:30:00.160ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
43132025-04-30T06:30:00.162ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdf5b63c-a84e-472c-8e13-8815c3a2f7a2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43142025-04-30T06:30:00.162ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
43152025-04-30T06:30:00.163ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff36c183-ed19-4d6a-9cce-bc449e3ccb22", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43162025-04-30T06:30:00.163ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
43172025-04-30T06:30:00.164ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21fccd41-1887-4611-93b6-89af143be737", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43182025-04-30T06:30:00.164ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
43192025-04-30T06:30:00.166ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c275014a-6ad8-4d3e-be95-a74f53f8bf39", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43202025-04-30T06:30:00.166ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
43212025-04-30T06:30:00.167ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c240274-736e-4fc0-b813-b802829c57e9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43222025-04-30T06:30:00.167ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
43232025-04-30T06:30:00.170ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d5b8a81-81b5-4e6f-b2e6-642cbd5ad9ee", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43242025-04-30T06:30:00.170ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
43252025-04-30T06:30:00.172ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "003c794c-1b0d-43aa-baf9-624e6ef2a8dd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43262025-04-30T06:30:00.172ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
43272025-04-30T06:30:00.174ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0baa6fd-923a-4d33-8e6f-63e2b5ee56a6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43282025-04-30T06:30:00.175ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
43292025-04-30T06:30:00.177ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "373e41e3-5349-42a2-a97c-8c860b6effab", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43302025-04-30T06:30:00.177ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
43312025-04-30T06:30:00.180ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8cf2cf0-ee33-4c87-a9ed-032be6406e07", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43322025-04-30T06:30:00.180ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
43332025-04-30T06:30:00.182ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af7b9e7f-0131-4a41-871e-148bfefdcc22", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43342025-04-30T06:30:00.182ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
43352025-04-30T06:30:00.183ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9923d62-ad51-47a6-af89-9431d08da7e8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43362025-04-30T06:30:00.183ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
43372025-04-30T06:30:00.184ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb5ebccf-6b95-499c-933c-2a3e2ce4a9f9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43382025-04-30T06:30:00.184ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
43392025-04-30T06:30:00.185ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27e62eca-78e7-4a28-8d77-8dc26d9f3f66", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43402025-04-30T06:30:00.185ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
43412025-04-30T06:30:00.186ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e399a6ba-50d7-473b-b7d2-d6e1fa99da8f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43422025-04-30T06:30:00.186ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
43432025-04-30T06:30:00.188ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67e4e53a-27d5-4dd4-82f9-4b029086503d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43442025-04-30T06:30:00.188ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
43452025-04-30T06:30:00.191ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21318785-b8ab-4676-9d6f-919e37a8edff", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43462025-04-30T06:30:00.191ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
43472025-04-30T06:30:00.195ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20d4ad45-d72a-4c1c-8a18-763e3bedc240", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43482025-04-30T06:30:00.195ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
43492025-04-30T06:30:00.197ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43502025-04-30T06:30:00.197ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43512025-04-30T06:30:00.197ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43522025-04-30T06:30:00.197ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
43532025-04-30T06:30:00.199ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43542025-04-30T06:30:00.202ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6126f908-254b-461d-afdb-730fbd704032", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43552025-04-30T06:30:00.202ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
43562025-04-30T06:30:00.203ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43572025-04-30T06:30:00.209ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43582025-04-30T06:30:00.209ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43592025-04-30T06:30:00.209ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
43602025-04-30T06:30:00.210ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43612025-04-30T06:30:00.213ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43622025-04-30T06:30:00.213ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
43632025-04-30T06:30:00.214ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43642025-04-30T06:30:00.217ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "596aab15-e9e1-4974-b01d-a12acd526b06", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43652025-04-30T06:30:00.217ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
43662025-04-30T06:30:00.220ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37e680ef-dedd-43eb-bddf-5c43786ef827", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43672025-04-30T06:30:00.220ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
43682025-04-30T06:30:00.226ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfa4007e-5a51-4b29-a388-058763084069", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43692025-04-30T06:30:00.226ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
43702025-04-30T06:30:00.228ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8771fd45-5932-4a97-bbba-748af53cc9ab", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43712025-04-30T06:30:00.228ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
43722025-04-30T06:30:00.229ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06ba7b5c-d00c-4949-870d-71b900d6c264", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43732025-04-30T06:30:00.231ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
43742025-04-30T06:30:00.232ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a0ef406-d940-4544-982e-90889e04934c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43752025-04-30T06:30:00.233ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
43762025-04-30T06:30:00.236ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19c09f1d-91a4-4a2a-8dbe-59683e4355ba", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43772025-04-30T06:30:00.236ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
43782025-04-30T06:30:00.240ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6800f43e-0ca5-4eed-a24d-5fabbba1f230", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43792025-04-30T06:30:00.241ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
43802025-04-30T06:30:00.243ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f509f2a-d711-4d92-864c-1d580c93dccc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43812025-04-30T06:30:00.243ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
43822025-04-30T06:30:00.248ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06110fb3-0ba3-4e2e-83e7-4fa4be3ca91d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43832025-04-30T06:30:00.249ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
43842025-04-30T06:30:00.249ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
43852025-04-30T06:30:00.254ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "359795f6-10b6-41d0-a294-ac6aea07885e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43862025-04-30T06:30:00.254ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
43872025-04-30T06:30:00.258ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f10977cf-0abc-4269-9fb4-54f85b00418f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43882025-04-30T06:30:00.258ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
43892025-04-30T06:30:00.259ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d9a8327-c3b2-4d6d-9d42-dd5df550ba24", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43902025-04-30T06:30:00.260ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
43912025-04-30T06:30:00.264ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e189176-b9e4-4173-9230-3343b93ec503", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43922025-04-30T06:30:00.264ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
43932025-04-30T06:30:00.265ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a3b4258-df55-447b-8a52-b2feed4b23ba", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43942025-04-30T06:30:00.266ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
43952025-04-30T06:30:00.268ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2d3e3bf-d87a-4516-a5ac-5fb2837bbc52", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43962025-04-30T06:30:00.268ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
43972025-04-30T06:30:00.270ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f18e107-abdf-4bfa-af47-1069e49896af", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
43982025-04-30T06:30:00.270ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
43992025-04-30T06:30:00.271ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76f4a947-9b73-4b49-a107-def70564d8ee", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
44002025-04-30T06:30:00.271ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44012025-04-30T06:30:00.271ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44022025-04-30T06:30:00.271ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44032025-04-30T06:30:00.271ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44042025-04-30T06:30:00.273ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44052025-04-30T06:30:00.275ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44062025-04-30T06:30:00.279ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44072025-04-30T06:30:00.279ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44082025-04-30T06:30:00.279ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44092025-04-30T06:30:00.280ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44102025-04-30T06:30:00.280ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44112025-04-30T06:30:00.280ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44122025-04-30T06:30:00.285ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44132025-04-30T06:30:00.289ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44142025-04-30T06:30:00.289ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44152025-04-30T06:30:00.289ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44162025-04-30T06:30:00.289ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44172025-04-30T06:30:00.359ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
44182025-04-30T06:30:00.359ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
44192025-04-30T06:30:00.360ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44202025-04-30T06:30:00.360ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44212025-04-30T06:30:00.360ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44222025-04-30T06:30:00.360ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
44232025-04-30T06:30:00.361ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44242025-04-30T06:30:00.361ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:56990 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
44252025-04-30T06:30:00.362ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:56990 req_id = ace9762d-3fca-4cdb-9988-e2f1bcd7b658 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44262025-04-30T06:30:00.363ZINFOtest_image_promotion (dropshot): request completed latency_us = 429 local_addr = [::1]:60070 method = GET remote_addr = [::1]:56990 req_id = ace9762d-3fca-4cdb-9988-e2f1bcd7b658 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44272025-04-30T06:30:00.363ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ace9762d-3fca-4cdb-9988-e2f1bcd7b658", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
44282025-04-30T06:30:00.366ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44292025-04-30T06:30:00.366ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44302025-04-30T06:30:00.366ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
44312025-04-30T06:30:00.371ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44322025-04-30T06:30:00.371ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44332025-04-30T06:30:00.371ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44342025-04-30T06:30:00.371ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44352025-04-30T06:30:00.372ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44362025-04-30T06:30:00.373ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44372025-04-30T06:30:00.385ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44382025-04-30T06:30:00.385ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44392025-04-30T06:30:00.385ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
44402025-04-30T06:30:00.386ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44412025-04-30T06:30:00.388ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44422025-04-30T06:30:00.392ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44432025-04-30T06:30:00.393ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44442025-04-30T06:30:00.393ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44452025-04-30T06:30:00.394ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44462025-04-30T06:30:00.395ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44472025-04-30T06:30:00.395ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
44482025-04-30T06:30:00.396ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44492025-04-30T06:30:00.461ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
44502025-04-30T06:30:00.461ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
44512025-04-30T06:30:00.462ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 443.990282ms iteration = 9 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
44522025-04-30T06:30:00.462ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 10 reason = Signaled
44532025-04-30T06:30:00.462ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44542025-04-30T06:30:00.462ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44552025-04-30T06:30:00.462ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44562025-04-30T06:30:00.462ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44572025-04-30T06:30:00.463ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44582025-04-30T06:30:00.467ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44592025-04-30T06:30:00.472ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44602025-04-30T06:30:00.472ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44612025-04-30T06:30:00.472ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44622025-04-30T06:30:00.473ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44632025-04-30T06:30:00.478ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
44642025-04-30T06:30:00.478ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
44652025-04-30T06:30:00.480ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ab e2 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55419 req_id = 16943f58-e959-4bcd-9cc8-4118f41e59ac
44662025-04-30T06:30:00.480ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44002,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55419 req_id = 16943f58-e959-4bcd-9cc8-4118f41e59ac
44672025-04-30T06:30:00.480ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
44682025-04-30T06:30:00.480ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
44692025-04-30T06:30:00.480ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55419 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 16943f58-e959-4bcd-9cc8-4118f41e59ac
44702025-04-30T06:30:00.481ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
44712025-04-30T06:30:00.481ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3465, tv_nsec: 280295379 } })
44722025-04-30T06:30:00.481ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
44732025-04-30T06:30:00.482ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 14 5f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ._..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53584 req_id = e0b0cd25-849c-4558-a7fb-138bab3fd289
44742025-04-30T06:30:00.482ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5215,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53584 req_id = e0b0cd25-849c-4558-a7fb-138bab3fd289
44752025-04-30T06:30:00.482ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
44762025-04-30T06:30:00.482ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53584 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e0b0cd25-849c-4558-a7fb-138bab3fd289
44772025-04-30T06:30:00.546ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
44782025-04-30T06:30:00.547ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44792025-04-30T06:30:00.547ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44802025-04-30T06:30:00.547ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44812025-04-30T06:30:00.547ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
44822025-04-30T06:30:00.548ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44832025-04-30T06:30:00.548ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
44842025-04-30T06:30:00.548ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44852025-04-30T06:30:00.550ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44862025-04-30T06:30:00.550ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44872025-04-30T06:30:00.550ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 62106544-08b3-4b9e-9a1b-d9d8020b7f12 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44882025-04-30T06:30:00.551ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 83 f0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61022 req_id = 37341b4f-65e0-4bf1-b0dc-218a8217c9d3
44892025-04-30T06:30:00.551ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33776,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61022 req_id = 37341b4f-65e0-4bf1-b0dc-218a8217c9d3
44902025-04-30T06:30:00.551ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
44912025-04-30T06:30:00.551ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33776, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61022 req_id = 37341b4f-65e0-4bf1-b0dc-218a8217c9d3
44922025-04-30T06:30:00.551ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 62106544-08b3-4b9e-9a1b-d9d8020b7f12 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44932025-04-30T06:30:00.551ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 62106544-08b3-4b9e-9a1b-d9d8020b7f12 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44942025-04-30T06:30:00.552ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 62106544-08b3-4b9e-9a1b-d9d8020b7f12 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44952025-04-30T06:30:00.555ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
44962025-04-30T06:30:00.555ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
44972025-04-30T06:30:00.555ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
44982025-04-30T06:30:00.555ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44992025-04-30T06:30:00.558ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45002025-04-30T06:30:00.558ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45012025-04-30T06:30:00.558ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45022025-04-30T06:30:00.558ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45032025-04-30T06:30:00.567ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45042025-04-30T06:30:00.569ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45052025-04-30T06:30:00.569ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
45062025-04-30T06:30:00.573ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45072025-04-30T06:30:00.578ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 27634 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 62106544-08b3-4b9e-9a1b-d9d8020b7f12 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45082025-04-30T06:30:00.580ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "62106544-08b3-4b9e-9a1b-d9d8020b7f12", "content-length": "405", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45092025-04-30T06:30:00.580ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
45102025-04-30T06:30:00.581ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45112025-04-30T06:30:00.581ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
45122025-04-30T06:30:00.582ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45132025-04-30T06:30:00.582ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 40b3c2b3-2c78-40fc-a9be-525914626214 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
45142025-04-30T06:30:00.583ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 40b3c2b3-2c78-40fc-a9be-525914626214 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
45152025-04-30T06:30:00.583ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 40b3c2b3-2c78-40fc-a9be-525914626214 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
45162025-04-30T06:30:00.584ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 40b3c2b3-2c78-40fc-a9be-525914626214 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
45172025-04-30T06:30:00.586ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8bba8a2-2623-45f2-b129-5f5ad2660459", "content-length": "835", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45182025-04-30T06:30:00.586ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
45192025-04-30T06:30:00.591ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 8343 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 40b3c2b3-2c78-40fc-a9be-525914626214 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
45202025-04-30T06:30:00.593ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "40b3c2b3-2c78-40fc-a9be-525914626214", "content-length": "29", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45212025-04-30T06:30:00.593ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45222025-04-30T06:30:00.593ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
45232025-04-30T06:30:00.594ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
45242025-04-30T06:30:00.595ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd752d03-7c97-42f6-b351-6e39339642d1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45252025-04-30T06:30:00.596ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
45262025-04-30T06:30:00.601ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7820756b-aea3-4f7f-ae03-82e95a016c1b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45272025-04-30T06:30:00.601ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
45282025-04-30T06:30:00.604ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "615af655-2b57-4259-a126-4b58eb7c9223", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45292025-04-30T06:30:00.604ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
45302025-04-30T06:30:00.617ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0760482-336c-4ae0-b162-287f11e629cd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45312025-04-30T06:30:00.617ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
45322025-04-30T06:30:00.619ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "994bccec-7f28-4838-a3d5-4da803d314b8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45332025-04-30T06:30:00.620ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
45342025-04-30T06:30:00.622ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2444ba35-7685-4ed0-9d24-73082d825c43", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45352025-04-30T06:30:00.622ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
45362025-04-30T06:30:00.624ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3afa12bc-a7bb-46e6-a42a-1de6766c8225", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45372025-04-30T06:30:00.624ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
45382025-04-30T06:30:00.627ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e244337f-ee92-4a42-9cd7-468730aca750", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45392025-04-30T06:30:00.627ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
45402025-04-30T06:30:00.627ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
45412025-04-30T06:30:00.628ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d8929c0-31aa-4c23-bb0e-f2805ddea2d8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45422025-04-30T06:30:00.628ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
45432025-04-30T06:30:00.629ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4151116a-95c6-4f8d-ad18-7bbc57af2458", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45442025-04-30T06:30:00.629ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
45452025-04-30T06:30:00.630ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aff5a7c7-5f2d-42c4-891c-4afaa0316d25", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45462025-04-30T06:30:00.630ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
45472025-04-30T06:30:00.632ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fe9257b-0123-49bd-9085-b6310be41e75", "content-length": "734", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45482025-04-30T06:30:00.632ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
45492025-04-30T06:30:00.634ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7000c88-8840-4b87-b157-5ed845155a77", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45502025-04-30T06:30:00.634ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
45512025-04-30T06:30:00.636ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5054a9d7-5a4f-4e59-ab24-a27e0a9eef7f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45522025-04-30T06:30:00.636ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
45532025-04-30T06:30:00.638ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "070eee7a-87af-4b21-be73-a863edb9a721", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45542025-04-30T06:30:00.638ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
45552025-04-30T06:30:00.640ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b9946a6-3ffe-4594-9a4e-0243957c6f11", "content-length": "170", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45562025-04-30T06:30:00.640ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
45572025-04-30T06:30:00.642ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "35f33d3c-2b19-40d2-8b2d-7f4e25200417", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45582025-04-30T06:30:00.642ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
45592025-04-30T06:30:00.645ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "435a10de-5edf-4cb4-b997-b021e42c3332", "content-length": "179", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45602025-04-30T06:30:00.645ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
45612025-04-30T06:30:00.645ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddd4ac27-c376-4ecb-88e5-27ab7ea767bb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45622025-04-30T06:30:00.646ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
45632025-04-30T06:30:00.648ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dc7c443-2568-45e1-a53a-cbe2a2b929f8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45642025-04-30T06:30:00.648ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
45652025-04-30T06:30:00.649ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "545bd1ef-cd01-4507-af69-aa6ee94b5b60", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45662025-04-30T06:30:00.649ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
45672025-04-30T06:30:00.651ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d467a16-97ad-4081-81c9-14f4762d577e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45682025-04-30T06:30:00.652ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
45692025-04-30T06:30:00.655ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32991bd0-8859-4956-b0aa-d7ea52495e3f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45702025-04-30T06:30:00.655ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
45712025-04-30T06:30:00.657ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3142b256-a2e8-48aa-915c-b60d88cb7a36", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45722025-04-30T06:30:00.657ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
45732025-04-30T06:30:00.660ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d4c8da0-60bd-4ae9-8a23-c75477639eb3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45742025-04-30T06:30:00.661ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
45752025-04-30T06:30:00.663ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "022b8971-bd32-4961-a079-6e146c54a07c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45762025-04-30T06:30:00.663ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
45772025-04-30T06:30:00.665ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae8cb822-46ff-4775-8c68-b9b229e734bd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45782025-04-30T06:30:00.665ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
45792025-04-30T06:30:00.669ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "936277c0-d25b-4414-b9ed-f1511d1a487d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45802025-04-30T06:30:00.669ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
45812025-04-30T06:30:00.675ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27fc80e0-b5f3-44f1-8b41-dac80f521fcb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45822025-04-30T06:30:00.677ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
45832025-04-30T06:30:00.678ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ec54ef0-edc2-418c-83d5-6c9e00d089e7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45842025-04-30T06:30:00.678ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
45852025-04-30T06:30:00.679ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45862025-04-30T06:30:00.679ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45872025-04-30T06:30:00.680ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45882025-04-30T06:30:00.680ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
45892025-04-30T06:30:00.682ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45902025-04-30T06:30:00.687ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5537f0bb-ba05-4e0d-a2d4-2db7ed973c72", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45912025-04-30T06:30:00.687ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
45922025-04-30T06:30:00.690ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45932025-04-30T06:30:00.696ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45942025-04-30T06:30:00.696ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45952025-04-30T06:30:00.696ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
45962025-04-30T06:30:00.696ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45972025-04-30T06:30:00.697ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34075938-edbd-4267-8726-91b92024fa6c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
45982025-04-30T06:30:00.697ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
45992025-04-30T06:30:00.701ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ebdc7cb-1789-41b0-9a77-5e83bc5e5b89", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
46002025-04-30T06:30:00.701ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
46012025-04-30T06:30:00.703ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46022025-04-30T06:30:00.703ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
46032025-04-30T06:30:00.703ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46042025-04-30T06:30:00.708ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38093ae3-6bdd-48ad-a129-6a2856245c9a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
46052025-04-30T06:30:00.708ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
46062025-04-30T06:30:00.717ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb468538-cc94-4340-8a8f-38a7323f975b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
46072025-04-30T06:30:00.717ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
46082025-04-30T06:30:00.725ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bf90879-f677-4a0f-ac7a-34f793c3cfb2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
46092025-04-30T06:30:00.726ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
46102025-04-30T06:30:00.729ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be040006-1e42-4820-bdfa-f3e8a1cae3ff", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
46112025-04-30T06:30:00.729ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
46122025-04-30T06:30:00.734ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74af3bef-7b45-4224-a927-67e542edc1fc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
46132025-04-30T06:30:00.735ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
46142025-04-30T06:30:00.738ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
46152025-04-30T06:30:00.740ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6aaab72f-41c2-44b2-bad5-1c7e0989dc21", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
46162025-04-30T06:30:00.741ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
46172025-04-30T06:30:00.744ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06fb4589-c71d-4732-b70d-28d3a5604252", "content-length": "178", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
46182025-04-30T06:30:00.745ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
46192025-04-30T06:30:00.746ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4775c81f-de22-4587-a123-62a8138a976c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
46202025-04-30T06:30:00.746ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
46212025-04-30T06:30:00.748ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b209c972-333b-4e07-8b49-094419fef4f7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
46222025-04-30T06:30:00.748ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
46232025-04-30T06:30:00.749ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaecb15e-8de2-4bdf-ab8f-8effde9965b0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
46242025-04-30T06:30:00.749ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
46252025-04-30T06:30:00.749ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46262025-04-30T06:30:00.750ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46272025-04-30T06:30:00.750ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
46282025-04-30T06:30:00.751ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
46292025-04-30T06:30:00.755ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f540e40-1841-42c1-ae87-611c197edc8b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:29:59 GMT"} })
46302025-04-30T06:30:00.756ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46312025-04-30T06:30:00.756ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46322025-04-30T06:30:00.756ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46332025-04-30T06:30:00.756ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46342025-04-30T06:30:00.757ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46352025-04-30T06:30:00.758ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
46362025-04-30T06:30:00.758ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
46372025-04-30T06:30:00.762ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
46382025-04-30T06:30:00.762ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46392025-04-30T06:30:00.763ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46402025-04-30T06:30:00.768ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46412025-04-30T06:30:00.768ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46422025-04-30T06:30:00.768ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46432025-04-30T06:30:00.768ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46442025-04-30T06:30:00.768ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46452025-04-30T06:30:00.769ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46462025-04-30T06:30:00.774ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46472025-04-30T06:30:00.779ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46482025-04-30T06:30:00.780ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46492025-04-30T06:30:00.780ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46502025-04-30T06:30:00.780ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46512025-04-30T06:30:00.853ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
46522025-04-30T06:30:00.853ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
46532025-04-30T06:30:00.853ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46542025-04-30T06:30:00.854ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46552025-04-30T06:30:00.854ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46562025-04-30T06:30:00.854ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
46572025-04-30T06:30:00.855ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46582025-04-30T06:30:00.855ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:54846 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
46592025-04-30T06:30:00.855ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
46602025-04-30T06:30:00.856ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:54846 req_id = 43315636-3536-44e5-aad6-b32bab7c4d79 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46612025-04-30T06:30:00.857ZINFOtest_image_promotion (dropshot): request completed latency_us = 867 local_addr = [::1]:60070 method = GET remote_addr = [::1]:54846 req_id = 43315636-3536-44e5-aad6-b32bab7c4d79 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46622025-04-30T06:30:00.858ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "43315636-3536-44e5-aad6-b32bab7c4d79", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
46632025-04-30T06:30:00.861ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46642025-04-30T06:30:00.861ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46652025-04-30T06:30:00.861ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
46662025-04-30T06:30:00.864ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46672025-04-30T06:30:00.864ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46682025-04-30T06:30:00.864ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46692025-04-30T06:30:00.864ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46702025-04-30T06:30:00.865ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46712025-04-30T06:30:00.866ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46722025-04-30T06:30:00.872ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46732025-04-30T06:30:00.872ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46742025-04-30T06:30:00.873ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
46752025-04-30T06:30:00.874ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46762025-04-30T06:30:00.882ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46772025-04-30T06:30:00.887ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
46782025-04-30T06:30:00.888ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46792025-04-30T06:30:00.888ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46802025-04-30T06:30:00.889ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46812025-04-30T06:30:00.893ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46822025-04-30T06:30:00.893ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
46832025-04-30T06:30:00.895ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46842025-04-30T06:30:00.952ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
46852025-04-30T06:30:00.953ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
46862025-04-30T06:30:00.953ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 491.051488ms iteration = 10 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
46872025-04-30T06:30:00.953ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 11 reason = Signaled
46882025-04-30T06:30:00.953ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46892025-04-30T06:30:00.953ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46902025-04-30T06:30:00.953ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46912025-04-30T06:30:00.953ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46922025-04-30T06:30:00.956ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46932025-04-30T06:30:00.958ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46942025-04-30T06:30:00.967ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
46952025-04-30T06:30:00.967ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46962025-04-30T06:30:00.968ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46972025-04-30T06:30:00.970ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46982025-04-30T06:30:00.976ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
46992025-04-30T06:30:00.979ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
47002025-04-30T06:30:00.979ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
47012025-04-30T06:30:00.988ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d5 6b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .k..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59260 req_id = bc1306a4-ea3f-4fe5-b8ca-ea8cc96022d6
47022025-04-30T06:30:00.988ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54635,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59260 req_id = bc1306a4-ea3f-4fe5-b8ca-ea8cc96022d6
47032025-04-30T06:30:00.989ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
47042025-04-30T06:30:00.989ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47052025-04-30T06:30:00.989ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59260 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = bc1306a4-ea3f-4fe5-b8ca-ea8cc96022d6
47062025-04-30T06:30:00.990ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3465, tv_nsec: 789709513 } })
47072025-04-30T06:30:00.990ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
47082025-04-30T06:30:00.994ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 45 5b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e E[..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57039 req_id = d4a13764-6270-4f92-8aef-cb250c74f8d2
47092025-04-30T06:30:00.994ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17755,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57039 req_id = d4a13764-6270-4f92-8aef-cb250c74f8d2
47102025-04-30T06:30:00.994ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47112025-04-30T06:30:00.995ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57039 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d4a13764-6270-4f92-8aef-cb250c74f8d2
47122025-04-30T06:30:01.117ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
47132025-04-30T06:30:01.125ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47142025-04-30T06:30:01.125ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47152025-04-30T06:30:01.125ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47162025-04-30T06:30:01.125ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
47172025-04-30T06:30:01.126ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47182025-04-30T06:30:01.132ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 60 c2 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c `............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60125 req_id = 6e611ce5-f4cf-4fb9-b42d-bd67d206b68c
47192025-04-30T06:30:01.132ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24770,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60125 req_id = 6e611ce5-f4cf-4fb9-b42d-bd67d206b68c
47202025-04-30T06:30:01.132ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
47212025-04-30T06:30:01.132ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24770, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60125 req_id = 6e611ce5-f4cf-4fb9-b42d-bd67d206b68c
47222025-04-30T06:30:01.133ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
47232025-04-30T06:30:01.133ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
47242025-04-30T06:30:01.133ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
47252025-04-30T06:30:01.136ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47262025-04-30T06:30:01.143ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47272025-04-30T06:30:01.143ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47282025-04-30T06:30:01.143ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
47292025-04-30T06:30:01.143ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47302025-04-30T06:30:01.150ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47312025-04-30T06:30:01.150ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
47322025-04-30T06:30:01.151ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47332025-04-30T06:30:01.159ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0edf9cf3-2c94-4a11-81ec-1ff9aca4bb40", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47342025-04-30T06:30:01.160ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
47352025-04-30T06:30:01.168ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59135f9a-c4b9-461c-abe2-16659fcc146c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47362025-04-30T06:30:01.168ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
47372025-04-30T06:30:01.172ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05bf68ef-cd12-44a2-b551-ca2432858e75", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47382025-04-30T06:30:01.173ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
47392025-04-30T06:30:01.176ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ae9fc7c-4d7c-4432-8953-6497d7125b0b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47402025-04-30T06:30:01.176ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
47412025-04-30T06:30:01.181ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfbddca5-7ab2-4217-8a04-6908020f7923", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47422025-04-30T06:30:01.182ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
47432025-04-30T06:30:01.186ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b0f445d-5c0d-4091-8402-054fd4cc7110", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47442025-04-30T06:30:01.187ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
47452025-04-30T06:30:01.195ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a9ca644-68f0-429c-866f-eb637e09abff", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47462025-04-30T06:30:01.195ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
47472025-04-30T06:30:01.197ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
47482025-04-30T06:30:01.202ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "79c73254-49d7-42c6-a01d-cf7685cb4de6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47492025-04-30T06:30:01.202ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
47502025-04-30T06:30:01.204ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80b1f3ac-8943-4afe-974d-4736e366e6f6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47512025-04-30T06:30:01.205ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
47522025-04-30T06:30:01.208ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93109b87-bd95-4bc6-85a9-f214f5f5d0fb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47532025-04-30T06:30:01.208ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
47542025-04-30T06:30:01.209ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b934d8bd-9e64-443b-8067-6f7d519c7053", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47552025-04-30T06:30:01.209ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
47562025-04-30T06:30:01.211ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fac15b9-3868-441e-a882-a2f361bb270d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47572025-04-30T06:30:01.211ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
47582025-04-30T06:30:01.213ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4efc1da-0d83-4aca-bf72-102610f209ab", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47592025-04-30T06:30:01.213ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
47602025-04-30T06:30:01.216ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7235fcd-6403-477a-acaa-e0c5cbf7833c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47612025-04-30T06:30:01.216ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
47622025-04-30T06:30:01.220ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5eb34c6-a3fb-4be4-8b26-94d3588aa75c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47632025-04-30T06:30:01.221ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
47642025-04-30T06:30:01.226ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "783d54e4-193f-42df-84f8-13caae56fa7f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47652025-04-30T06:30:01.227ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
47662025-04-30T06:30:01.230ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9c6e1ab-f84d-4f4c-8a38-c726e2443780", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47672025-04-30T06:30:01.230ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
47682025-04-30T06:30:01.234ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8780a20-8659-4e38-bf73-f6da92f835bc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47692025-04-30T06:30:01.234ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
47702025-04-30T06:30:01.237ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97c0f521-78e8-4ba9-b768-5f8b85360e98", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47712025-04-30T06:30:01.237ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
47722025-04-30T06:30:01.238ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "285cd5e7-6ea7-4e69-a0a6-03ef3cbc20f3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47732025-04-30T06:30:01.238ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
47742025-04-30T06:30:01.241ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de2b2c94-d91e-41e9-8b21-873df740afe6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47752025-04-30T06:30:01.242ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
47762025-04-30T06:30:01.244ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12f3d874-53a4-4acb-9d4e-f2ee89cb5e6b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47772025-04-30T06:30:01.244ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
47782025-04-30T06:30:01.245ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55869fb9-4c29-4363-a59e-065c1e2a8e20", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47792025-04-30T06:30:01.245ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
47802025-04-30T06:30:01.246ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e28c532-da56-4e1c-9258-378b95ac13b9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47812025-04-30T06:30:01.246ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
47822025-04-30T06:30:01.249ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47832025-04-30T06:30:01.249ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47842025-04-30T06:30:01.249ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47852025-04-30T06:30:01.249ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
47862025-04-30T06:30:01.250ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47872025-04-30T06:30:01.250ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a6963fb-6d5f-4bc4-bee1-e04d99ce1072", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47882025-04-30T06:30:01.250ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
47892025-04-30T06:30:01.253ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47902025-04-30T06:30:01.264ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47912025-04-30T06:30:01.264ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47922025-04-30T06:30:01.264ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
47932025-04-30T06:30:01.264ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47942025-04-30T06:30:01.265ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a486b3e-b557-47e2-b06f-080b050f0070", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
47952025-04-30T06:30:01.265ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
47962025-04-30T06:30:01.268ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47972025-04-30T06:30:01.268ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
47982025-04-30T06:30:01.269ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47992025-04-30T06:30:01.270ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5444cfc-90af-44eb-8b3c-e22ddc1feb8f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48002025-04-30T06:30:01.270ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
48012025-04-30T06:30:01.274ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12a43713-1689-49b7-a3f2-91e0e664349d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48022025-04-30T06:30:01.274ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
48032025-04-30T06:30:01.280ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0127062e-2f3b-427f-a87f-1b9b3adfc4e5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48042025-04-30T06:30:01.280ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
48052025-04-30T06:30:01.289ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b072708-1f02-46c8-a7b5-7bc0aecd96dc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48062025-04-30T06:30:01.290ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
48072025-04-30T06:30:01.296ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "609a71c7-8d47-4f58-a668-6c0c7e414f76", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48082025-04-30T06:30:01.296ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
48092025-04-30T06:30:01.298ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ff63739-2e38-42c3-a7db-85d166f972c2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48102025-04-30T06:30:01.298ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
48112025-04-30T06:30:01.300ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59e66fd1-118f-4bbc-9629-d8c45b8b3798", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48122025-04-30T06:30:01.300ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
48132025-04-30T06:30:01.301ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db79c994-c66f-4777-b9a9-6980e0ff50b5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48142025-04-30T06:30:01.301ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
48152025-04-30T06:30:01.303ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6aef92a9-f1ad-44db-bc74-ee2f9e2559cb", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48162025-04-30T06:30:01.303ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
48172025-04-30T06:30:01.308ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
48182025-04-30T06:30:01.323ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d995446-ca8a-43c8-961f-f278e84769a5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48192025-04-30T06:30:01.323ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
48202025-04-30T06:30:01.329ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a446a2ad-3f8e-4aad-98b1-6e6243cc1033", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48212025-04-30T06:30:01.330ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
48222025-04-30T06:30:01.333ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e26f6cf-0836-42a5-841b-1347165756e1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48232025-04-30T06:30:01.333ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
48242025-04-30T06:30:01.337ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b82d8fa-6a77-4d85-a7b8-e7166272dfd0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48252025-04-30T06:30:01.337ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
48262025-04-30T06:30:01.342ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d19b9ff-0e45-4f59-9561-6b450071db4c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48272025-04-30T06:30:01.343ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
48282025-04-30T06:30:01.347ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75cedcb0-ffa5-4754-8170-364d7ba6f02f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48292025-04-30T06:30:01.347ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
48302025-04-30T06:30:01.348ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35ebb263-d278-4ffc-b6c0-56de16ee38e5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48312025-04-30T06:30:01.348ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
48322025-04-30T06:30:01.350ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44a00793-dcca-4503-8e4a-770e8f27cd50", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48332025-04-30T06:30:01.351ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
48342025-04-30T06:30:01.356ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9755f8b1-dff4-444c-8eca-ce6c8ee8cbfe", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48352025-04-30T06:30:01.356ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
48362025-04-30T06:30:01.358ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce72aacf-cace-4b33-872b-c9d679432deb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48372025-04-30T06:30:01.358ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48382025-04-30T06:30:01.359ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48392025-04-30T06:30:01.359ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48402025-04-30T06:30:01.359ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48412025-04-30T06:30:01.360ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48422025-04-30T06:30:01.361ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48432025-04-30T06:30:01.361ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48442025-04-30T06:30:01.362ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48452025-04-30T06:30:01.362ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
48462025-04-30T06:30:01.365ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48472025-04-30T06:30:01.368ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48482025-04-30T06:30:01.380ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48492025-04-30T06:30:01.380ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48502025-04-30T06:30:01.381ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48512025-04-30T06:30:01.381ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48522025-04-30T06:30:01.381ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48532025-04-30T06:30:01.381ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48542025-04-30T06:30:01.384ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48552025-04-30T06:30:01.391ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48562025-04-30T06:30:01.391ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48572025-04-30T06:30:01.391ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
48582025-04-30T06:30:01.391ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48592025-04-30T06:30:01.394ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48602025-04-30T06:30:01.399ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48612025-04-30T06:30:01.399ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48622025-04-30T06:30:01.399ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48632025-04-30T06:30:01.400ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48642025-04-30T06:30:01.402ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48652025-04-30T06:30:01.402ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
48662025-04-30T06:30:01.402ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48672025-04-30T06:30:01.510ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
48682025-04-30T06:30:01.510ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
48692025-04-30T06:30:01.512ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:35767 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
48702025-04-30T06:30:01.512ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:35767 req_id = 506ee15d-8078-4ba7-8a91-03dc0ffd7083 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
48712025-04-30T06:30:01.513ZINFOtest_image_promotion (dropshot): request completed latency_us = 811 local_addr = [::1]:60070 method = GET remote_addr = [::1]:35767 req_id = 506ee15d-8078-4ba7-8a91-03dc0ffd7083 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
48722025-04-30T06:30:01.514ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "506ee15d-8078-4ba7-8a91-03dc0ffd7083", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
48732025-04-30T06:30:01.517ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
48742025-04-30T06:30:01.517ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
48752025-04-30T06:30:01.517ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
48762025-04-30T06:30:01.522ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
48772025-04-30T06:30:01.522ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48782025-04-30T06:30:01.522ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48792025-04-30T06:30:01.522ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48802025-04-30T06:30:01.523ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48812025-04-30T06:30:01.530ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48822025-04-30T06:30:01.539ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
48832025-04-30T06:30:01.539ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48842025-04-30T06:30:01.539ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48852025-04-30T06:30:01.539ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48862025-04-30T06:30:01.540ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48872025-04-30T06:30:01.540ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48882025-04-30T06:30:01.546ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48892025-04-30T06:30:01.546ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48902025-04-30T06:30:01.547ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48912025-04-30T06:30:01.547ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48922025-04-30T06:30:01.584ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
48932025-04-30T06:30:01.593ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
48942025-04-30T06:30:01.593ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
48952025-04-30T06:30:01.593ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 640.021311ms iteration = 11 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
48962025-04-30T06:30:01.593ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 12 reason = Signaled
48972025-04-30T06:30:01.593ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
48982025-04-30T06:30:01.593ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48992025-04-30T06:30:01.593ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49002025-04-30T06:30:01.593ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49012025-04-30T06:30:01.595ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49022025-04-30T06:30:01.599ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49032025-04-30T06:30:01.603ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49042025-04-30T06:30:01.603ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49052025-04-30T06:30:01.603ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49062025-04-30T06:30:01.603ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49072025-04-30T06:30:01.607ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
49082025-04-30T06:30:01.607ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
49092025-04-30T06:30:01.609ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 86 6d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .m..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53016 req_id = c4ed0e73-0673-4069-983e-d2f52bfc0b2f
49102025-04-30T06:30:01.609ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34413,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53016 req_id = c4ed0e73-0673-4069-983e-d2f52bfc0b2f
49112025-04-30T06:30:01.609ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
49122025-04-30T06:30:01.609ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49132025-04-30T06:30:01.609ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53016 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c4ed0e73-0673-4069-983e-d2f52bfc0b2f
49142025-04-30T06:30:01.609ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3466, tv_nsec: 408791694 } })
49152025-04-30T06:30:01.610ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
49162025-04-30T06:30:01.610ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7f bc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59510 req_id = 979720f7-dc94-46c8-b57b-97d813f121a7
49172025-04-30T06:30:01.610ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32700,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59510 req_id = 979720f7-dc94-46c8-b57b-97d813f121a7
49182025-04-30T06:30:01.610ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49192025-04-30T06:30:01.610ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59510 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 979720f7-dc94-46c8-b57b-97d813f121a7
49202025-04-30T06:30:01.655ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
49212025-04-30T06:30:01.655ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49222025-04-30T06:30:01.655ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49232025-04-30T06:30:01.655ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49242025-04-30T06:30:01.655ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
49252025-04-30T06:30:01.656ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49262025-04-30T06:30:01.657ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 10 6f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .o..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61482 req_id = c2f6eb2e-d42d-4fa5-ad7d-7ee3eb171d01
49272025-04-30T06:30:01.657ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4207,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61482 req_id = c2f6eb2e-d42d-4fa5-ad7d-7ee3eb171d01
49282025-04-30T06:30:01.657ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
49292025-04-30T06:30:01.657ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4207, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61482 req_id = c2f6eb2e-d42d-4fa5-ad7d-7ee3eb171d01
49302025-04-30T06:30:01.658ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
49312025-04-30T06:30:01.658ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
49322025-04-30T06:30:01.658ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
49332025-04-30T06:30:01.659ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49342025-04-30T06:30:01.668ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49352025-04-30T06:30:01.668ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49362025-04-30T06:30:01.668ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
49372025-04-30T06:30:01.669ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49382025-04-30T06:30:01.669ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "25eeb0b4-954a-42b1-9a65-b502d89bd713", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49392025-04-30T06:30:01.669ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
49402025-04-30T06:30:01.672ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49412025-04-30T06:30:01.672ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
49422025-04-30T06:30:01.674ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49432025-04-30T06:30:01.675ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49309187-6a77-46eb-b274-f835e4c18969", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49442025-04-30T06:30:01.676ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
49452025-04-30T06:30:01.683ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d494fce1-22a3-45c8-9c83-b6054a562a0b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49462025-04-30T06:30:01.683ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
49472025-04-30T06:30:01.690ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f56319f8-18f0-40f3-b90c-1c28ec80d040", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49482025-04-30T06:30:01.690ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
49492025-04-30T06:30:01.692ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d99f4815-04fc-41eb-99ad-397774eb7f8b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49502025-04-30T06:30:01.692ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
49512025-04-30T06:30:01.695ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3edb781e-e2ed-409b-91a2-96e0de5a7ba7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49522025-04-30T06:30:01.695ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
49532025-04-30T06:30:01.697ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee6ba0da-e865-4357-826a-00e956ac82e0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49542025-04-30T06:30:01.697ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
49552025-04-30T06:30:01.701ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97c2c2c7-e27c-424d-895d-2682e09901d3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49562025-04-30T06:30:01.702ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
49572025-04-30T06:30:01.703ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5da8c7d8-5a9f-4186-8c43-28068a1bcdbc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49582025-04-30T06:30:01.703ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
49592025-04-30T06:30:01.706ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20b1c294-3f0b-4843-974d-2f182c77c897", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49602025-04-30T06:30:01.706ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
49612025-04-30T06:30:01.710ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7de080b-9d87-4a98-a884-1067a8c9075f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49622025-04-30T06:30:01.710ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
49632025-04-30T06:30:01.711ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7d05563-8aff-48ed-8107-ea551c6c152d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49642025-04-30T06:30:01.711ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
49652025-04-30T06:30:01.712ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
49662025-04-30T06:30:01.714ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5c66ad3-b42a-4840-b240-2233287779f7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49672025-04-30T06:30:01.714ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
49682025-04-30T06:30:01.716ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32360093-1802-4dd9-9424-0f3c1dc891f7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49692025-04-30T06:30:01.717ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
49702025-04-30T06:30:01.718ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b4ba709-6bb1-4e47-92ad-81802d474d5a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49712025-04-30T06:30:01.718ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
49722025-04-30T06:30:01.721ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e7fbdfe-3c92-4779-aa0a-a6f8f12de2da", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49732025-04-30T06:30:01.721ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
49742025-04-30T06:30:01.725ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a555692a-6b54-45f0-bd07-d25a2ee4a135", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49752025-04-30T06:30:01.725ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
49762025-04-30T06:30:01.737ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc3c32f3-b360-444a-8f8f-af0955851ea0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49772025-04-30T06:30:01.738ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
49782025-04-30T06:30:01.742ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ee96138-df2b-4dbf-bf07-2907b3d59caa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49792025-04-30T06:30:01.742ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
49802025-04-30T06:30:01.743ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4d5dab3-9f5e-4f7f-aef1-37fa3c45f8ff", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49812025-04-30T06:30:01.743ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
49822025-04-30T06:30:01.744ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a11c9e82-cbef-45e4-88a7-8c8ae1a21b76", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49832025-04-30T06:30:01.744ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
49842025-04-30T06:30:01.745ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c7f6677-c96b-4bfc-8424-88b478b25062", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49852025-04-30T06:30:01.745ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
49862025-04-30T06:30:01.749ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f173300-8ce9-4381-988f-0c8442c28adf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49872025-04-30T06:30:01.749ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
49882025-04-30T06:30:01.751ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
49892025-04-30T06:30:01.751ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
49902025-04-30T06:30:01.751ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
49912025-04-30T06:30:01.753ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
49922025-04-30T06:30:01.759ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
49932025-04-30T06:30:01.759ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
49942025-04-30T06:30:01.762ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81afdc74-c0e5-49ec-b853-409011fbc51b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
49952025-04-30T06:30:01.763ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
49962025-04-30T06:30:01.763ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49972025-04-30T06:30:01.763ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49982025-04-30T06:30:01.763ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49992025-04-30T06:30:01.763ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
50002025-04-30T06:30:01.764ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50012025-04-30T06:30:01.766ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
50022025-04-30T06:30:01.766ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50032025-04-30T06:30:01.769ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50042025-04-30T06:30:01.778ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50052025-04-30T06:30:01.778ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50062025-04-30T06:30:01.778ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
50072025-04-30T06:30:01.778ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50082025-04-30T06:30:01.779ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "536b2ca7-95a1-4fc7-bb2e-db351d38ec5d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50092025-04-30T06:30:01.780ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
50102025-04-30T06:30:01.783ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50112025-04-30T06:30:01.783ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
50122025-04-30T06:30:01.785ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50132025-04-30T06:30:01.789ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6f40d46-e401-4c0d-8481-d3154738ddbd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50142025-04-30T06:30:01.789ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
50152025-04-30T06:30:01.794ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f70a2f4-c301-425d-be1a-4d1815ef02c9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50162025-04-30T06:30:01.794ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
50172025-04-30T06:30:01.798ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dd209f4-7ad3-49f0-a33b-2a9b3623a632", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50182025-04-30T06:30:01.798ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
50192025-04-30T06:30:01.805ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b04898f-140a-4e67-af43-5ebacae7f43a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50202025-04-30T06:30:01.805ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
50212025-04-30T06:30:01.807ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6401db66-f6ce-4285-8516-13e43d1a39e7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50222025-04-30T06:30:01.807ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
50232025-04-30T06:30:01.809ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28db0abf-9009-4af3-950b-e7e8380a4bf4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50242025-04-30T06:30:01.809ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
50252025-04-30T06:30:01.810ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d103c962-d03e-4379-ac2c-a879ef108ed0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50262025-04-30T06:30:01.811ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
50272025-04-30T06:30:01.812ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da230bdb-f17d-4ec8-97a9-247ba231241a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50282025-04-30T06:30:01.812ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
50292025-04-30T06:30:01.814ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "efd087d0-64c0-481a-ac9a-0f9ed568a84f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50302025-04-30T06:30:01.814ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
50312025-04-30T06:30:01.816ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afe1c66e-af2b-4311-8ff5-d79fca2e6b87", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50322025-04-30T06:30:01.817ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
50332025-04-30T06:30:01.818ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
50342025-04-30T06:30:01.822ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc156834-690e-4b2f-8e18-6b31dd664104", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50352025-04-30T06:30:01.822ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
50362025-04-30T06:30:01.824ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b4d9a98-fc22-4c65-94e2-9b607668a536", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50372025-04-30T06:30:01.825ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
50382025-04-30T06:30:01.830ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "214b1d1a-60c9-4183-9d48-ee498cc212f6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50392025-04-30T06:30:01.830ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
50402025-04-30T06:30:01.837ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0baafca9-d660-46f0-b5cf-a2001b2bfb6a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50412025-04-30T06:30:01.838ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
50422025-04-30T06:30:01.845ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c58906f-56e3-4181-b6a1-a6ffdc9c5ec7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50432025-04-30T06:30:01.845ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
50442025-04-30T06:30:01.851ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78d0e57f-5a0e-4385-aaa6-865b199b3901", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50452025-04-30T06:30:01.851ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
50462025-04-30T06:30:01.854ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
50472025-04-30T06:30:01.854ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58d73def-c3a5-4287-979c-166e1afeb993", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:00 GMT"} })
50482025-04-30T06:30:01.854ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
50492025-04-30T06:30:01.858ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cc9c155-6b3d-4cd6-afda-932c86a4099c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
50502025-04-30T06:30:01.858ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
50512025-04-30T06:30:01.858ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e137005-02f7-4929-a978-5cb67e3e4f1a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
50522025-04-30T06:30:01.858ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
50532025-04-30T06:30:01.860ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10bea018-7b1e-4a8d-9309-ba1dc1ad2285", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
50542025-04-30T06:30:01.860ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50552025-04-30T06:30:01.860ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50562025-04-30T06:30:01.860ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50572025-04-30T06:30:01.860ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50582025-04-30T06:30:01.861ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50592025-04-30T06:30:01.865ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50602025-04-30T06:30:01.872ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
50612025-04-30T06:30:01.872ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50622025-04-30T06:30:01.872ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50632025-04-30T06:30:01.872ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50642025-04-30T06:30:01.872ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50652025-04-30T06:30:01.873ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50662025-04-30T06:30:01.875ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50672025-04-30T06:30:01.875ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50682025-04-30T06:30:01.875ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50692025-04-30T06:30:01.875ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
50702025-04-30T06:30:01.875ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50712025-04-30T06:30:01.877ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50722025-04-30T06:30:01.881ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
50732025-04-30T06:30:01.882ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50742025-04-30T06:30:01.882ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50752025-04-30T06:30:01.882ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50762025-04-30T06:30:01.883ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50772025-04-30T06:30:01.888ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50782025-04-30T06:30:01.888ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50792025-04-30T06:30:01.888ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
50802025-04-30T06:30:01.889ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50812025-04-30T06:30:01.955ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
50822025-04-30T06:30:01.955ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
50832025-04-30T06:30:01.956ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:51867 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
50842025-04-30T06:30:01.959ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:51867 req_id = aa0e591c-2831-4aad-a994-bbcc81d167ed sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
50852025-04-30T06:30:01.960ZINFOtest_image_promotion (dropshot): request completed latency_us = 594 local_addr = [::1]:60070 method = GET remote_addr = [::1]:51867 req_id = aa0e591c-2831-4aad-a994-bbcc81d167ed response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
50862025-04-30T06:30:01.961ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50872025-04-30T06:30:01.961ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
50882025-04-30T06:30:01.962ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50892025-04-30T06:30:01.963ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa0e591c-2831-4aad-a994-bbcc81d167ed", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
50902025-04-30T06:30:01.965ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
50912025-04-30T06:30:01.965ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
50922025-04-30T06:30:01.965ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
50932025-04-30T06:30:01.968ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
50942025-04-30T06:30:01.968ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50952025-04-30T06:30:01.968ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50962025-04-30T06:30:01.968ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50972025-04-30T06:30:01.970ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50982025-04-30T06:30:01.985ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50992025-04-30T06:30:01.991ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51002025-04-30T06:30:01.991ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51012025-04-30T06:30:01.991ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51022025-04-30T06:30:01.994ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51032025-04-30T06:30:02.050ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
51042025-04-30T06:30:02.050ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
51052025-04-30T06:30:02.050ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 457.202505ms iteration = 12 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
51062025-04-30T06:30:02.050ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 13 reason = Signaled
51072025-04-30T06:30:02.050ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51082025-04-30T06:30:02.050ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51092025-04-30T06:30:02.051ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51102025-04-30T06:30:02.051ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51112025-04-30T06:30:02.051ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51122025-04-30T06:30:02.056ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51132025-04-30T06:30:02.063ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51142025-04-30T06:30:02.063ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51152025-04-30T06:30:02.063ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51162025-04-30T06:30:02.063ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51172025-04-30T06:30:02.064ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
51182025-04-30T06:30:02.071ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
51192025-04-30T06:30:02.071ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
51202025-04-30T06:30:02.072ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 67 d7 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 g............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53738 req_id = 612049ce-8521-4adc-a4b6-fff064d23d6c
51212025-04-30T06:30:02.073ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26583,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53738 req_id = 612049ce-8521-4adc-a4b6-fff064d23d6c
51222025-04-30T06:30:02.073ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
51232025-04-30T06:30:02.073ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51242025-04-30T06:30:02.073ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53738 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 612049ce-8521-4adc-a4b6-fff064d23d6c
51252025-04-30T06:30:02.073ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3466, tv_nsec: 872891843 } })
51262025-04-30T06:30:02.074ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
51272025-04-30T06:30:02.074ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9f 1b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58868 req_id = 133cafb4-200f-44e7-92b5-543e2ec79818
51282025-04-30T06:30:02.074ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40731,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58868 req_id = 133cafb4-200f-44e7-92b5-543e2ec79818
51292025-04-30T06:30:02.075ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51302025-04-30T06:30:02.075ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58868 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 133cafb4-200f-44e7-92b5-543e2ec79818
51312025-04-30T06:30:02.176ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
51322025-04-30T06:30:02.177ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51332025-04-30T06:30:02.177ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51342025-04-30T06:30:02.177ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51352025-04-30T06:30:02.177ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
51362025-04-30T06:30:02.178ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51372025-04-30T06:30:02.178ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a6 5a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .Z..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60311 req_id = ca4093ad-9f6d-406b-9b89-8af89461a9c2
51382025-04-30T06:30:02.179ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42586,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60311 req_id = ca4093ad-9f6d-406b-9b89-8af89461a9c2
51392025-04-30T06:30:02.180ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
51402025-04-30T06:30:02.180ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42586, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60311 req_id = ca4093ad-9f6d-406b-9b89-8af89461a9c2
51412025-04-30T06:30:02.181ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
51422025-04-30T06:30:02.181ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
51432025-04-30T06:30:02.181ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
51442025-04-30T06:30:02.182ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51452025-04-30T06:30:02.192ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51462025-04-30T06:30:02.192ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51472025-04-30T06:30:02.192ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
51482025-04-30T06:30:02.193ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51492025-04-30T06:30:02.198ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c6d69a2-1e19-4fb1-b47c-6414134286fa", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51502025-04-30T06:30:02.198ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
51512025-04-30T06:30:02.199ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51522025-04-30T06:30:02.199ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
51532025-04-30T06:30:02.200ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51542025-04-30T06:30:02.202ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "726b12e2-674c-49d8-ade2-81a9d3ad0d84", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51552025-04-30T06:30:02.203ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
51562025-04-30T06:30:02.209ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a9aa20b-0d30-4710-a5a1-78398a5ff759", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51572025-04-30T06:30:02.209ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
51582025-04-30T06:30:02.213ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b6646ac-b122-4cb3-9051-021c8282f44f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51592025-04-30T06:30:02.215ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
51602025-04-30T06:30:02.224ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "903b52db-4721-4e14-af30-a252679a847e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51612025-04-30T06:30:02.224ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
51622025-04-30T06:30:02.229ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a6a4ba7-aa9a-4008-8e96-2999e4acf993", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51632025-04-30T06:30:02.229ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
51642025-04-30T06:30:02.233ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "796460e8-8f77-4a4e-8601-1101ce2d7b1d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51652025-04-30T06:30:02.233ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
51662025-04-30T06:30:02.234ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
51672025-04-30T06:30:02.237ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c52b448-d64d-4322-aabc-4d23a374e99d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51682025-04-30T06:30:02.237ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
51692025-04-30T06:30:02.238ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe55817f-0c77-4317-87ed-76bff015d628", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51702025-04-30T06:30:02.239ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
51712025-04-30T06:30:02.240ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b84ff79-4172-4da9-99c8-0db3be86c65f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51722025-04-30T06:30:02.240ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
51732025-04-30T06:30:02.241ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "050d8c22-55a2-4469-8ddb-5e02034c30d4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51742025-04-30T06:30:02.241ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
51752025-04-30T06:30:02.242ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c60d478-ee88-4d5f-abf5-b4f3e16d4bfd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51762025-04-30T06:30:02.242ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
51772025-04-30T06:30:02.244ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "392973a4-2c2e-405b-ba5e-858b085416a0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51782025-04-30T06:30:02.244ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
51792025-04-30T06:30:02.246ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "555dc34f-5526-4166-b2c3-95b6f6d0efed", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51802025-04-30T06:30:02.246ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
51812025-04-30T06:30:02.248ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35a8bea8-bada-4357-b763-59d8dc264dc1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51822025-04-30T06:30:02.249ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
51832025-04-30T06:30:02.250ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3ad45c1-5534-4235-a48d-b7f8d05e1798", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51842025-04-30T06:30:02.251ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
51852025-04-30T06:30:02.253ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9806c3e7-8531-425d-a71c-4c33b4171818", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51862025-04-30T06:30:02.255ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
51872025-04-30T06:30:02.259ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d2d9540-c037-4782-ad5c-de7b2628d5e7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51882025-04-30T06:30:02.259ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
51892025-04-30T06:30:02.262ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d713c237-5aeb-4563-9bd8-39cb15c0be13", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51902025-04-30T06:30:02.262ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
51912025-04-30T06:30:02.263ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72280a15-d42c-4478-9e4c-87aa5f6bbcc1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51922025-04-30T06:30:02.264ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
51932025-04-30T06:30:02.265ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f357586-f687-463e-8b86-2e277f93915e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51942025-04-30T06:30:02.265ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
51952025-04-30T06:30:02.268ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e74d1a24-1fc2-4d99-b419-3b7123c7b94a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51962025-04-30T06:30:02.268ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
51972025-04-30T06:30:02.270ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04001ae4-8b45-46cd-88aa-7bf340399f33", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
51982025-04-30T06:30:02.270ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
51992025-04-30T06:30:02.273ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1829e53-a1b1-4daa-8d9d-c0e6af4f2447", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52002025-04-30T06:30:02.273ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
52012025-04-30T06:30:02.279ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1843f8a6-cc2c-4fb7-af2e-8751f04098cf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52022025-04-30T06:30:02.279ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
52032025-04-30T06:30:02.284ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c6086b3-b936-47b3-b528-8e03f5b286aa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52042025-04-30T06:30:02.284ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
52052025-04-30T06:30:02.285ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52062025-04-30T06:30:02.285ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52072025-04-30T06:30:02.285ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52082025-04-30T06:30:02.285ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
52092025-04-30T06:30:02.287ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52102025-04-30T06:30:02.289ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52112025-04-30T06:30:02.301ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52122025-04-30T06:30:02.301ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52132025-04-30T06:30:02.301ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
52142025-04-30T06:30:02.302ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52152025-04-30T06:30:02.305ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b489d25-0b72-4b12-b585-ce6862d52a80", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52162025-04-30T06:30:02.305ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
52172025-04-30T06:30:02.306ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52182025-04-30T06:30:02.306ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
52192025-04-30T06:30:02.309ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52202025-04-30T06:30:02.313ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b02b821-3878-4aef-83c6-ec90531a0c8a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52212025-04-30T06:30:02.313ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
52222025-04-30T06:30:02.319ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eef46325-4862-4a00-864f-3534bbc826cc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52232025-04-30T06:30:02.319ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
52242025-04-30T06:30:02.329ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82b89c7f-1804-405f-8a56-8c1742baf996", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52252025-04-30T06:30:02.329ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
52262025-04-30T06:30:02.331ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff47dd83-ad91-4dbc-acb0-9f6d95db7449", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52272025-04-30T06:30:02.331ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
52282025-04-30T06:30:02.333ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7751cc90-0536-4cfb-9143-957d920f5afb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52292025-04-30T06:30:02.333ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
52302025-04-30T06:30:02.335ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "564aac97-ba6c-4834-a806-5e241e95d39d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52312025-04-30T06:30:02.335ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
52322025-04-30T06:30:02.336ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9132010-fb01-4539-a90e-c9521059c9f8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52332025-04-30T06:30:02.336ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
52342025-04-30T06:30:02.339ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69fe7435-bbb7-474e-842e-8077fde2a9f3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52352025-04-30T06:30:02.339ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
52362025-04-30T06:30:02.342ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "104fbef9-661b-4032-9a00-46893173c02a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52372025-04-30T06:30:02.342ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
52382025-04-30T06:30:02.346ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
52392025-04-30T06:30:02.349ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ca7ee91-1d1b-4363-8c9d-cfd8a5e8931f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52402025-04-30T06:30:02.350ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
52412025-04-30T06:30:02.355ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a103732-62df-4bff-8b08-97de67208c14", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52422025-04-30T06:30:02.356ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
52432025-04-30T06:30:02.359ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a19eed3a-cf2d-48fe-bc3a-84d5a3b8c65f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52442025-04-30T06:30:02.359ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
52452025-04-30T06:30:02.362ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d3ab061-8c29-4607-bd51-712b89dc603a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52462025-04-30T06:30:02.362ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
52472025-04-30T06:30:02.366ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a4b2d46-e680-4fae-a5d1-f5f187d748ba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52482025-04-30T06:30:02.366ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
52492025-04-30T06:30:02.367ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b74f3914-805f-4e86-b1e1-1fc03bc16bb3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52502025-04-30T06:30:02.367ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
52512025-04-30T06:30:02.368ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1d5d029-7e4b-44a4-aea4-fd31d84c3697", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52522025-04-30T06:30:02.368ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
52532025-04-30T06:30:02.370ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2938a12c-42c1-4add-bcb3-0f3bec0322b2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52542025-04-30T06:30:02.370ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
52552025-04-30T06:30:02.372ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "073b1150-8043-40e6-97a7-e543086977f0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52562025-04-30T06:30:02.372ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
52572025-04-30T06:30:02.372ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52582025-04-30T06:30:02.372ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52592025-04-30T06:30:02.372ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52602025-04-30T06:30:02.373ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52612025-04-30T06:30:02.375ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52622025-04-30T06:30:02.382ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
52632025-04-30T06:30:02.382ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
52642025-04-30T06:30:02.382ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52652025-04-30T06:30:02.382ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52662025-04-30T06:30:02.382ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52672025-04-30T06:30:02.383ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52682025-04-30T06:30:02.388ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52692025-04-30T06:30:02.398ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
52702025-04-30T06:30:02.398ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52712025-04-30T06:30:02.398ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52722025-04-30T06:30:02.399ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52732025-04-30T06:30:02.399ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52742025-04-30T06:30:02.399ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52752025-04-30T06:30:02.399ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52762025-04-30T06:30:02.400ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
52772025-04-30T06:30:02.400ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52782025-04-30T06:30:02.403ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52792025-04-30T06:30:02.414ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52802025-04-30T06:30:02.414ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52812025-04-30T06:30:02.414ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
52822025-04-30T06:30:02.415ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52832025-04-30T06:30:02.496ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
52842025-04-30T06:30:02.497ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
52852025-04-30T06:30:02.500ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:61974 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
52862025-04-30T06:30:02.500ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:61974 req_id = 7f80c56e-b32c-458b-9edc-f9d652d176a6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
52872025-04-30T06:30:02.501ZINFOtest_image_promotion (dropshot): request completed latency_us = 474 local_addr = [::1]:60070 method = GET remote_addr = [::1]:61974 req_id = 7f80c56e-b32c-458b-9edc-f9d652d176a6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
52882025-04-30T06:30:02.501ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52892025-04-30T06:30:02.501ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
52902025-04-30T06:30:02.503ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52912025-04-30T06:30:02.504ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f80c56e-b32c-458b-9edc-f9d652d176a6", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
52922025-04-30T06:30:02.516ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
52932025-04-30T06:30:02.517ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
52942025-04-30T06:30:02.517ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
52952025-04-30T06:30:02.523ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
52962025-04-30T06:30:02.523ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52972025-04-30T06:30:02.523ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52982025-04-30T06:30:02.523ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52992025-04-30T06:30:02.523ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53002025-04-30T06:30:02.524ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
53012025-04-30T06:30:02.525ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53022025-04-30T06:30:02.528ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = a91b46de-c0c7-4dfd-a187-1c031868d7a5 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53032025-04-30T06:30:02.528ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = a91b46de-c0c7-4dfd-a187-1c031868d7a5 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53042025-04-30T06:30:02.528ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = a91b46de-c0c7-4dfd-a187-1c031868d7a5 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53052025-04-30T06:30:02.528ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = a91b46de-c0c7-4dfd-a187-1c031868d7a5 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53062025-04-30T06:30:02.528ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53072025-04-30T06:30:02.533ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53082025-04-30T06:30:02.533ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53092025-04-30T06:30:02.533ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53102025-04-30T06:30:02.534ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53112025-04-30T06:30:02.538ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53122025-04-30T06:30:02.538ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53132025-04-30T06:30:02.538ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 11458 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = a91b46de-c0c7-4dfd-a187-1c031868d7a5 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53142025-04-30T06:30:02.541ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "a91b46de-c0c7-4dfd-a187-1c031868d7a5", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
53152025-04-30T06:30:02.541ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
53162025-04-30T06:30:02.545ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fb6fe1d9-a4bd-4887-b431-e58b192f98b0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
53172025-04-30T06:30:02.546ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fb6fe1d9-a4bd-4887-b431-e58b192f98b0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
53182025-04-30T06:30:02.546ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fb6fe1d9-a4bd-4887-b431-e58b192f98b0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
53192025-04-30T06:30:02.547ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fb6fe1d9-a4bd-4887-b431-e58b192f98b0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
53202025-04-30T06:30:02.547ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53212025-04-30T06:30:02.548ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53222025-04-30T06:30:02.548ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53232025-04-30T06:30:02.548ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53242025-04-30T06:30:02.569ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 19009 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fb6fe1d9-a4bd-4887-b431-e58b192f98b0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
53252025-04-30T06:30:02.569ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb6fe1d9-a4bd-4887-b431-e58b192f98b0", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
53262025-04-30T06:30:02.569ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53272025-04-30T06:30:02.569ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
53282025-04-30T06:30:02.569ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
53292025-04-30T06:30:02.595ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
53302025-04-30T06:30:02.646ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53312025-04-30T06:30:02.646ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53322025-04-30T06:30:02.646ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53332025-04-30T06:30:02.646ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
53342025-04-30T06:30:02.647ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53352025-04-30T06:30:02.651ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53362025-04-30T06:30:02.660ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53372025-04-30T06:30:02.660ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53382025-04-30T06:30:02.660ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
53392025-04-30T06:30:02.660ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53402025-04-30T06:30:02.701ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53412025-04-30T06:30:02.701ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
53422025-04-30T06:30:02.702ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53432025-04-30T06:30:02.702ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
53442025-04-30T06:30:02.703ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
53452025-04-30T06:30:02.703ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 652.7586ms iteration = 13 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
53462025-04-30T06:30:02.703ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 14 reason = Signaled
53472025-04-30T06:30:02.703ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53482025-04-30T06:30:02.703ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53492025-04-30T06:30:02.703ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53502025-04-30T06:30:02.703ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53512025-04-30T06:30:02.707ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53522025-04-30T06:30:02.709ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53532025-04-30T06:30:02.717ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53542025-04-30T06:30:02.717ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53552025-04-30T06:30:02.717ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53562025-04-30T06:30:02.717ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53572025-04-30T06:30:02.724ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
53582025-04-30T06:30:02.724ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
53592025-04-30T06:30:02.725ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ff 8d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56435 req_id = 77a79c63-10fa-486e-bee9-47b7e262e618
53602025-04-30T06:30:02.725ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65421,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56435 req_id = 77a79c63-10fa-486e-bee9-47b7e262e618
53612025-04-30T06:30:02.725ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
53622025-04-30T06:30:02.726ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53632025-04-30T06:30:02.726ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56435 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 77a79c63-10fa-486e-bee9-47b7e262e618
53642025-04-30T06:30:02.726ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3467, tv_nsec: 525622767 } })
53652025-04-30T06:30:02.726ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
53662025-04-30T06:30:02.727ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9f 17 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55838 req_id = 6f8d9c2a-c6e9-46ac-a781-5f649553107e
53672025-04-30T06:30:02.727ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40727,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55838 req_id = 6f8d9c2a-c6e9-46ac-a781-5f649553107e
53682025-04-30T06:30:02.727ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53692025-04-30T06:30:02.727ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55838 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6f8d9c2a-c6e9-46ac-a781-5f649553107e
53702025-04-30T06:30:02.779ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
53712025-04-30T06:30:02.780ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
53722025-04-30T06:30:02.780ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
53732025-04-30T06:30:02.780ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
53742025-04-30T06:30:02.781ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
53752025-04-30T06:30:02.782ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2d 28 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c -(..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52998 req_id = ae890c7d-ff3b-4464-bd08-06a17643ea53
53762025-04-30T06:30:02.782ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11560,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52998 req_id = ae890c7d-ff3b-4464-bd08-06a17643ea53
53772025-04-30T06:30:02.783ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
53782025-04-30T06:30:02.783ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11560, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52998 req_id = ae890c7d-ff3b-4464-bd08-06a17643ea53
53792025-04-30T06:30:02.783ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
53802025-04-30T06:30:02.783ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
53812025-04-30T06:30:02.783ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
53822025-04-30T06:30:02.786ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee70b930-4595-44b7-87c3-f9942cda779a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
53832025-04-30T06:30:02.786ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
53842025-04-30T06:30:02.787ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
53852025-04-30T06:30:02.787ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
53862025-04-30T06:30:02.790ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "812e6e5b-d6bd-48dd-b6c8-4da65bedca7d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
53872025-04-30T06:30:02.790ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
53882025-04-30T06:30:02.791ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
53892025-04-30T06:30:02.791ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
53902025-04-30T06:30:02.793ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6dde59f-edde-4c88-bb52-3fd47dfd4f0c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
53912025-04-30T06:30:02.793ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
53922025-04-30T06:30:02.796ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6faa60c5-39cc-4f22-9608-64ccf169d9b9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
53932025-04-30T06:30:02.797ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
53942025-04-30T06:30:02.800ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d0cfe29-38f4-4c70-b7b4-6892d5e9d3be", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
53952025-04-30T06:30:02.800ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
53962025-04-30T06:30:02.802ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7933466-ba1d-42c5-bacd-566f167f72d0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
53972025-04-30T06:30:02.803ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
53982025-04-30T06:30:02.806ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c20cbe4-b339-44e7-aaac-dcc85cbfe9ac", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
53992025-04-30T06:30:02.806ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
54002025-04-30T06:30:02.812ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e55f253b-d9e0-4502-a101-c4edf57712da", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54012025-04-30T06:30:02.812ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
54022025-04-30T06:30:02.813ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9082e7f-adc3-4c61-9ae1-6164b7c38865", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54032025-04-30T06:30:02.814ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
54042025-04-30T06:30:02.815ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d89441ac-54bc-4e9b-8dc6-344088914fe7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54052025-04-30T06:30:02.816ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
54062025-04-30T06:30:02.819ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97613ff0-3b8a-40de-b9e7-dea58338a246", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54072025-04-30T06:30:02.820ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
54082025-04-30T06:30:02.821ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
54092025-04-30T06:30:02.823ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4c46fe9-a170-4f40-90e6-9683adc3ce36", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54102025-04-30T06:30:02.823ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
54112025-04-30T06:30:02.824ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55094695-857f-4216-a878-17c4620dc33c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54122025-04-30T06:30:02.825ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
54132025-04-30T06:30:02.830ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1710171-480d-4964-85a7-ce94b8716ec6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54142025-04-30T06:30:02.830ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
54152025-04-30T06:30:02.832ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b851a961-2c52-4eb0-9e04-e3f578796bf2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54162025-04-30T06:30:02.833ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
54172025-04-30T06:30:02.835ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bca7a0d-d2a0-4403-81e1-07b7403e47c5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54182025-04-30T06:30:02.835ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
54192025-04-30T06:30:02.839ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cd8271c-3a21-45b6-97fe-e6abec3409ee", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54202025-04-30T06:30:02.839ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
54212025-04-30T06:30:02.844ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13527c61-f157-4de5-a24b-892a412e5079", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54222025-04-30T06:30:02.845ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
54232025-04-30T06:30:02.849ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "859e1c1e-d3f5-4993-ac66-6d330868092c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54242025-04-30T06:30:02.850ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
54252025-04-30T06:30:02.852ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d60e40bf-b534-4f25-8e05-af76f6210bde", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54262025-04-30T06:30:02.852ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
54272025-04-30T06:30:02.855ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed6fdefd-ea90-43ba-b7d0-fc61791de89e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54282025-04-30T06:30:02.855ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
54292025-04-30T06:30:02.856ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38d907ec-c3c1-4669-b8af-19fadf6e8c62", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:01 GMT"} })
54302025-04-30T06:30:02.856ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
54312025-04-30T06:30:02.858ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9effa20e-763d-4748-a4ff-41eb22050bdb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54322025-04-30T06:30:02.859ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
54332025-04-30T06:30:02.861ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9177d7b5-f91e-41e4-8b3d-d9eeb6714f38", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54342025-04-30T06:30:02.861ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
54352025-04-30T06:30:02.864ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bf6f0c5-45d1-4541-a313-da1411855171", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54362025-04-30T06:30:02.864ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
54372025-04-30T06:30:02.867ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b101b9c6-1b9e-4db8-ac87-e17e7e24a004", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54382025-04-30T06:30:02.867ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
54392025-04-30T06:30:02.869ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42817afa-43cc-40bf-81f0-75abd973ad39", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54402025-04-30T06:30:02.870ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
54412025-04-30T06:30:02.872ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54422025-04-30T06:30:02.872ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54432025-04-30T06:30:02.872ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54442025-04-30T06:30:02.872ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
54452025-04-30T06:30:02.874ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54462025-04-30T06:30:02.877ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "736240d8-e219-4339-acc0-a823af25dd8c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54472025-04-30T06:30:02.877ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
54482025-04-30T06:30:02.878ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54492025-04-30T06:30:02.885ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54502025-04-30T06:30:02.885ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54512025-04-30T06:30:02.885ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
54522025-04-30T06:30:02.886ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54532025-04-30T06:30:02.887ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
54542025-04-30T06:30:02.889ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54552025-04-30T06:30:02.889ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
54562025-04-30T06:30:02.890ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54572025-04-30T06:30:02.894ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1152035a-7ced-411d-bce6-b89e4656a8bf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54582025-04-30T06:30:02.894ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
54592025-04-30T06:30:02.900ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcce0372-1d1a-45eb-a16f-41db181e0ed9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54602025-04-30T06:30:02.900ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
54612025-04-30T06:30:02.905ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0d12ba3-a2ae-456c-a473-b6ae0eff5194", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54622025-04-30T06:30:02.906ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
54632025-04-30T06:30:02.908ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02de29b3-0f95-4512-8475-5175e9d7c001", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54642025-04-30T06:30:02.908ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
54652025-04-30T06:30:02.910ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ca424eb-25ec-4288-88df-074218985b02", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54662025-04-30T06:30:02.910ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
54672025-04-30T06:30:02.913ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1979b42b-c45b-4011-a1f0-758444cb104b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54682025-04-30T06:30:02.913ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
54692025-04-30T06:30:02.917ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d85b9a3-d774-4835-a4da-3d39d86199d9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54702025-04-30T06:30:02.917ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
54712025-04-30T06:30:02.922ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41622b5c-8090-4c79-8ace-67ee63d7b5f1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54722025-04-30T06:30:02.922ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
54732025-04-30T06:30:02.927ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2c313c0-b02c-4261-a9d3-44afc3f7f402", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54742025-04-30T06:30:02.927ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
54752025-04-30T06:30:02.928ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
54762025-04-30T06:30:02.933ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01a5c740-dc04-4ba3-9629-3bc774411b5a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54772025-04-30T06:30:02.934ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
54782025-04-30T06:30:02.946ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a6b1ee3-ed5f-44f9-ac27-71b7c7fd4a4c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54792025-04-30T06:30:02.946ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
54802025-04-30T06:30:02.963ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "114c23d8-85ce-46ea-8156-f96c05fdbab4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54812025-04-30T06:30:02.963ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
54822025-04-30T06:30:02.980ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54832025-04-30T06:30:02.981ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54842025-04-30T06:30:02.981ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54852025-04-30T06:30:02.981ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
54862025-04-30T06:30:02.982ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54872025-04-30T06:30:02.993ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54882025-04-30T06:30:03.000ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54892025-04-30T06:30:03.001ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54902025-04-30T06:30:03.001ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
54912025-04-30T06:30:03.001ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54922025-04-30T06:30:03.004ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54932025-04-30T06:30:03.004ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
54942025-04-30T06:30:03.005ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54952025-04-30T06:30:03.009ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4a55036-d24a-40fc-a83c-086b2ac916da", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54962025-04-30T06:30:03.009ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
54972025-04-30T06:30:03.015ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "330de47c-a0e2-47c0-b1b9-90ebaf69a7e0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
54982025-04-30T06:30:03.016ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
54992025-04-30T06:30:03.017ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65cf6918-a6c6-42c7-b88c-4982e34c050d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
55002025-04-30T06:30:03.017ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
55012025-04-30T06:30:03.020ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "941f6766-f06f-435a-b468-46f3af8bfd60", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
55022025-04-30T06:30:03.020ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
55032025-04-30T06:30:03.021ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8462c05f-901d-4208-9bf8-476584d13fc3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
55042025-04-30T06:30:03.021ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55052025-04-30T06:30:03.021ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55062025-04-30T06:30:03.021ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55072025-04-30T06:30:03.021ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55082025-04-30T06:30:03.022ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55092025-04-30T06:30:03.026ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55102025-04-30T06:30:03.030ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55112025-04-30T06:30:03.030ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55122025-04-30T06:30:03.030ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55132025-04-30T06:30:03.030ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55142025-04-30T06:30:03.030ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55152025-04-30T06:30:03.030ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55162025-04-30T06:30:03.034ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55172025-04-30T06:30:03.041ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55182025-04-30T06:30:03.041ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55192025-04-30T06:30:03.041ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55202025-04-30T06:30:03.042ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55212025-04-30T06:30:03.109ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
55222025-04-30T06:30:03.109ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
55232025-04-30T06:30:03.112ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:48108 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
55242025-04-30T06:30:03.113ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:48108 req_id = 6ad6557f-0828-4c10-8df4-e559b9c26728 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55252025-04-30T06:30:03.113ZINFOtest_image_promotion (dropshot): request completed latency_us = 492 local_addr = [::1]:60070 method = GET remote_addr = [::1]:48108 req_id = 6ad6557f-0828-4c10-8df4-e559b9c26728 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55262025-04-30T06:30:03.113ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
55272025-04-30T06:30:03.114ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ad6557f-0828-4c10-8df4-e559b9c26728", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
55282025-04-30T06:30:03.117ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55292025-04-30T06:30:03.117ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55302025-04-30T06:30:03.117ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
55312025-04-30T06:30:03.120ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55322025-04-30T06:30:03.120ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55332025-04-30T06:30:03.120ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55342025-04-30T06:30:03.120ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55352025-04-30T06:30:03.121ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55362025-04-30T06:30:03.126ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55372025-04-30T06:30:03.131ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55382025-04-30T06:30:03.131ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55392025-04-30T06:30:03.131ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55402025-04-30T06:30:03.132ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55412025-04-30T06:30:03.165ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55422025-04-30T06:30:03.165ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55432025-04-30T06:30:03.165ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55442025-04-30T06:30:03.165ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
55452025-04-30T06:30:03.166ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55462025-04-30T06:30:03.172ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55472025-04-30T06:30:03.180ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55482025-04-30T06:30:03.180ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55492025-04-30T06:30:03.180ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
55502025-04-30T06:30:03.182ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55512025-04-30T06:30:03.201ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
55522025-04-30T06:30:03.201ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
55532025-04-30T06:30:03.201ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 498.03912ms iteration = 14 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
55542025-04-30T06:30:03.201ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 15 reason = Signaled
55552025-04-30T06:30:03.201ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55562025-04-30T06:30:03.201ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55572025-04-30T06:30:03.201ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55582025-04-30T06:30:03.202ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55592025-04-30T06:30:03.202ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55602025-04-30T06:30:03.205ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55612025-04-30T06:30:03.205ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
55622025-04-30T06:30:03.209ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55632025-04-30T06:30:03.211ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55642025-04-30T06:30:03.219ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55652025-04-30T06:30:03.220ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55662025-04-30T06:30:03.220ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55672025-04-30T06:30:03.220ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55682025-04-30T06:30:03.224ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
55692025-04-30T06:30:03.224ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
55702025-04-30T06:30:03.226ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 82 90 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58590 req_id = 1487404e-7fdc-4f40-b9bd-5f5415a1f3d7
55712025-04-30T06:30:03.226ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33424,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58590 req_id = 1487404e-7fdc-4f40-b9bd-5f5415a1f3d7
55722025-04-30T06:30:03.226ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
55732025-04-30T06:30:03.226ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55742025-04-30T06:30:03.226ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58590 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1487404e-7fdc-4f40-b9bd-5f5415a1f3d7
55752025-04-30T06:30:03.226ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3468, tv_nsec: 25708605 } })
55762025-04-30T06:30:03.226ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
55772025-04-30T06:30:03.227ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 82 9e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59133 req_id = 97dd2c80-0d7c-4b76-9a34-760a96f1b019
55782025-04-30T06:30:03.227ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33438,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59133 req_id = 97dd2c80-0d7c-4b76-9a34-760a96f1b019
55792025-04-30T06:30:03.228ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55802025-04-30T06:30:03.228ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59133 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 97dd2c80-0d7c-4b76-9a34-760a96f1b019
55812025-04-30T06:30:03.329ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
55822025-04-30T06:30:03.330ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 98 83 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61172 req_id = cf7d7082-665a-4aa3-a4e8-2698f7634d14
55832025-04-30T06:30:03.330ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39043,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61172 req_id = cf7d7082-665a-4aa3-a4e8-2698f7634d14
55842025-04-30T06:30:03.331ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
55852025-04-30T06:30:03.331ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39043, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61172 req_id = cf7d7082-665a-4aa3-a4e8-2698f7634d14
55862025-04-30T06:30:03.331ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
55872025-04-30T06:30:03.331ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
55882025-04-30T06:30:03.331ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
55892025-04-30T06:30:03.337ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1774ab3-89b5-441d-8a85-174b60337755", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
55902025-04-30T06:30:03.338ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
55912025-04-30T06:30:03.341ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b030cd42-7258-4d5d-86ac-a0668e6d97c5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
55922025-04-30T06:30:03.341ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
55932025-04-30T06:30:03.346ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49283c85-d238-410b-9c35-4dc3d58f27e1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
55942025-04-30T06:30:03.346ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
55952025-04-30T06:30:03.354ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c6695e1-e5c2-4e78-b3f4-87225cba44b5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
55962025-04-30T06:30:03.354ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
55972025-04-30T06:30:03.354ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
55982025-04-30T06:30:03.360ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05514816-7e8c-4d58-a209-a85a658b8910", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
55992025-04-30T06:30:03.360ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
56002025-04-30T06:30:03.366ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1c93b48-ee4e-4747-9fbc-c71cfc545df7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56012025-04-30T06:30:03.366ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
56022025-04-30T06:30:03.370ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0a07652-5cd6-4e79-8709-adcccb0b5cdc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56032025-04-30T06:30:03.370ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
56042025-04-30T06:30:03.370ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba171f2e-6e9a-4cac-8653-392550ef1364", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56052025-04-30T06:30:03.371ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
56062025-04-30T06:30:03.372ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b6188d0-22f8-4a33-b488-90480c44deab", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56072025-04-30T06:30:03.372ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
56082025-04-30T06:30:03.373ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ebc2720-8ecb-4c35-8c1c-f36f30b69a15", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56092025-04-30T06:30:03.373ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
56102025-04-30T06:30:03.376ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88be7cc2-d5d0-4049-ba66-f89420cd99d1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56112025-04-30T06:30:03.376ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
56122025-04-30T06:30:03.377ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f850dd7d-64cd-4430-b396-6e1f383c071c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56132025-04-30T06:30:03.378ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
56142025-04-30T06:30:03.380ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b1ee943-21b8-415e-9c81-2699bcce4ebd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56152025-04-30T06:30:03.380ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
56162025-04-30T06:30:03.383ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df7f5f07-68c1-4ad8-a712-c4baf842b2e1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56172025-04-30T06:30:03.383ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
56182025-04-30T06:30:03.385ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09e409ce-be0a-49ee-9880-539b7ea90812", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56192025-04-30T06:30:03.386ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
56202025-04-30T06:30:03.429ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56212025-04-30T06:30:03.429ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56222025-04-30T06:30:03.429ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56232025-04-30T06:30:03.429ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
56242025-04-30T06:30:03.430ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56252025-04-30T06:30:03.435ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c381ee36-7c8c-4ec0-a523-a1f03f37bebc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56262025-04-30T06:30:03.436ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
56272025-04-30T06:30:03.436ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56282025-04-30T06:30:03.442ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56292025-04-30T06:30:03.442ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56302025-04-30T06:30:03.442ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
56312025-04-30T06:30:03.443ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56322025-04-30T06:30:03.445ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2a6b52d-7b1c-4bfc-8201-8c463adad56d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56332025-04-30T06:30:03.445ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
56342025-04-30T06:30:03.447ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56352025-04-30T06:30:03.447ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
56362025-04-30T06:30:03.447ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56372025-04-30T06:30:03.449ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "239306af-a727-49af-bd7b-f9bd548c16b0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56382025-04-30T06:30:03.449ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
56392025-04-30T06:30:03.453ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "842001a9-95c1-4d5c-8cb4-cb92357f6ca4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56402025-04-30T06:30:03.453ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
56412025-04-30T06:30:03.454ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c3cbdf0-a5c1-42a5-a0a3-76110cc14b81", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56422025-04-30T06:30:03.455ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
56432025-04-30T06:30:03.458ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43f582a2-87ba-4230-8eff-b4488a8131a6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56442025-04-30T06:30:03.458ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
56452025-04-30T06:30:03.459ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "49005c47-0710-49fe-a3a5-bd01988a5897", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56462025-04-30T06:30:03.459ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
56472025-04-30T06:30:03.460ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b939c7bb-68b8-407a-a7d8-d6d89dade5f1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56482025-04-30T06:30:03.460ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
56492025-04-30T06:30:03.462ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "73a649cf-a3f0-492f-a595-438b11338222", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56502025-04-30T06:30:03.462ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
56512025-04-30T06:30:03.466ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f3bdfe0-c8df-4840-b3f8-132ab2f4c0ab", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56522025-04-30T06:30:03.467ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
56532025-04-30T06:30:03.473ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a478c926-25aa-4cbd-ac3a-9e093f6c617c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56542025-04-30T06:30:03.474ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
56552025-04-30T06:30:03.479ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d117b352-8e69-425f-939f-c082a2e92274", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56562025-04-30T06:30:03.479ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
56572025-04-30T06:30:03.483ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05259ff9-9fd1-4413-9014-9d2730bcbf0c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56582025-04-30T06:30:03.484ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
56592025-04-30T06:30:03.488ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18ebc88b-031e-4c76-8077-605398e782df", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56602025-04-30T06:30:03.488ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
56612025-04-30T06:30:03.489ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
56622025-04-30T06:30:03.492ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b1a617f-7479-46c1-80bd-d32a7aa31636", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56632025-04-30T06:30:03.492ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
56642025-04-30T06:30:03.494ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "733b5ebc-4148-4f3a-ace4-981c891f64ab", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56652025-04-30T06:30:03.494ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
56662025-04-30T06:30:03.495ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "afd1f283-171d-4ab6-a07d-fb0d1aefdfeb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56672025-04-30T06:30:03.495ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
56682025-04-30T06:30:03.497ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc3aefc2-2c99-4165-ae53-9268f48f8ea8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56692025-04-30T06:30:03.497ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
56702025-04-30T06:30:03.498ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4e3f704-7e36-4304-887c-717be85a34c4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56712025-04-30T06:30:03.498ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
56722025-04-30T06:30:03.500ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc9e0fd0-fa99-444d-b16c-004311bb5f37", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56732025-04-30T06:30:03.500ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
56742025-04-30T06:30:03.502ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b597d01-6c98-47ed-ad63-97b3d74f93b8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56752025-04-30T06:30:03.502ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
56762025-04-30T06:30:03.504ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cadbc08a-6b5c-4427-8ba6-361d9a43838e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56772025-04-30T06:30:03.504ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
56782025-04-30T06:30:03.507ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f9f0156-ce2d-4633-a659-0294c2c67d6a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56792025-04-30T06:30:03.507ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
56802025-04-30T06:30:03.510ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bee8ec1-ee52-4464-9fe9-e53608bbec58", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56812025-04-30T06:30:03.511ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
56822025-04-30T06:30:03.518ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f932d88f-9441-4db7-aead-3a8c7c6493cd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56832025-04-30T06:30:03.519ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
56842025-04-30T06:30:03.523ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7c9a0ca-e196-452d-97e6-ee261308a131", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56852025-04-30T06:30:03.523ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
56862025-04-30T06:30:03.524ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2ed0e8c-4085-4f8a-a997-f5c4fa1c5eec", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56872025-04-30T06:30:03.524ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
56882025-04-30T06:30:03.528ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c5c98df-5134-42c2-8dcc-36c0e7bdee47", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56892025-04-30T06:30:03.528ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
56902025-04-30T06:30:03.530ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7ae2f52-52cf-432f-b6dd-ca7ff7f288ea", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56912025-04-30T06:30:03.531ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
56922025-04-30T06:30:03.532ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d02bdcd-df71-47ae-b8b6-a7a17310239a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
56932025-04-30T06:30:03.533ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
56942025-04-30T06:30:03.533ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56952025-04-30T06:30:03.533ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56962025-04-30T06:30:03.534ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56972025-04-30T06:30:03.534ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56982025-04-30T06:30:03.538ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56992025-04-30T06:30:03.538ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57002025-04-30T06:30:03.538ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57012025-04-30T06:30:03.544ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57022025-04-30T06:30:03.544ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57032025-04-30T06:30:03.544ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57042025-04-30T06:30:03.544ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57052025-04-30T06:30:03.544ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57062025-04-30T06:30:03.545ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57072025-04-30T06:30:03.545ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57082025-04-30T06:30:03.546ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57092025-04-30T06:30:03.546ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57102025-04-30T06:30:03.546ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
57112025-04-30T06:30:03.547ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57122025-04-30T06:30:03.549ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57132025-04-30T06:30:03.550ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57142025-04-30T06:30:03.550ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57152025-04-30T06:30:03.551ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57162025-04-30T06:30:03.551ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57172025-04-30T06:30:03.556ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57182025-04-30T06:30:03.556ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57192025-04-30T06:30:03.556ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57202025-04-30T06:30:03.557ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57212025-04-30T06:30:03.558ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57222025-04-30T06:30:03.568ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57232025-04-30T06:30:03.568ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57242025-04-30T06:30:03.568ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
57252025-04-30T06:30:03.570ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57262025-04-30T06:30:03.651ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
57272025-04-30T06:30:03.652ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
57282025-04-30T06:30:03.654ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:55953 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
57292025-04-30T06:30:03.655ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57302025-04-30T06:30:03.655ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
57312025-04-30T06:30:03.656ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57322025-04-30T06:30:03.657ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:55953 req_id = 0d1c6172-8066-490f-b091-c77d88a91d47 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57332025-04-30T06:30:03.657ZINFOtest_image_promotion (dropshot): request completed latency_us = 876 local_addr = [::1]:60070 method = GET remote_addr = [::1]:55953 req_id = 0d1c6172-8066-490f-b091-c77d88a91d47 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57342025-04-30T06:30:03.658ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d1c6172-8066-490f-b091-c77d88a91d47", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
57352025-04-30T06:30:03.665ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57362025-04-30T06:30:03.665ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57372025-04-30T06:30:03.665ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
57382025-04-30T06:30:03.669ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57392025-04-30T06:30:03.669ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57402025-04-30T06:30:03.669ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57412025-04-30T06:30:03.669ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57422025-04-30T06:30:03.670ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57432025-04-30T06:30:03.676ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57442025-04-30T06:30:03.683ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57452025-04-30T06:30:03.683ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57462025-04-30T06:30:03.683ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57472025-04-30T06:30:03.684ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57482025-04-30T06:30:03.717ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
57492025-04-30T06:30:03.717ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
57502025-04-30T06:30:03.718ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
57512025-04-30T06:30:03.718ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 516.617707ms iteration = 15 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
57522025-04-30T06:30:03.718ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 16 reason = Signaled
57532025-04-30T06:30:03.718ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57542025-04-30T06:30:03.718ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57552025-04-30T06:30:03.718ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57562025-04-30T06:30:03.718ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57572025-04-30T06:30:03.719ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57582025-04-30T06:30:03.727ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57592025-04-30T06:30:03.733ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57602025-04-30T06:30:03.733ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57612025-04-30T06:30:03.733ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57622025-04-30T06:30:03.734ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57632025-04-30T06:30:03.741ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
57642025-04-30T06:30:03.741ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
57652025-04-30T06:30:03.741ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 64 ea 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 d............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51237 req_id = 6bd5ba98-a4bc-4ffc-b2c1-2bd6a04b527c
57662025-04-30T06:30:03.741ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25834,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51237 req_id = 6bd5ba98-a4bc-4ffc-b2c1-2bd6a04b527c
57672025-04-30T06:30:03.741ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
57682025-04-30T06:30:03.741ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
57692025-04-30T06:30:03.741ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51237 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6bd5ba98-a4bc-4ffc-b2c1-2bd6a04b527c
57702025-04-30T06:30:03.742ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3468, tv_nsec: 541173104 } })
57712025-04-30T06:30:03.742ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
57722025-04-30T06:30:03.742ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e9 ba 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58333 req_id = 8707f7a4-8142-460f-85f2-903f4e612773
57732025-04-30T06:30:03.742ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59834,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58333 req_id = 8707f7a4-8142-460f-85f2-903f4e612773
57742025-04-30T06:30:03.743ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
57752025-04-30T06:30:03.745ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58333 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8707f7a4-8142-460f-85f2-903f4e612773
57762025-04-30T06:30:03.795ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
57772025-04-30T06:30:03.796ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57782025-04-30T06:30:03.796ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57792025-04-30T06:30:03.796ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57802025-04-30T06:30:03.796ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
57812025-04-30T06:30:03.797ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57822025-04-30T06:30:03.797ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57832025-04-30T06:30:03.797ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57842025-04-30T06:30:03.798ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
57852025-04-30T06:30:03.800ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
57862025-04-30T06:30:03.801ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 29 95 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c )............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63878 req_id = 4a5ddc0b-7d88-45d3-a274-87ad9cf60610
57872025-04-30T06:30:03.801ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10645,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63878 req_id = 4a5ddc0b-7d88-45d3-a274-87ad9cf60610
57882025-04-30T06:30:03.801ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
57892025-04-30T06:30:03.801ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10645, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63878 req_id = 4a5ddc0b-7d88-45d3-a274-87ad9cf60610
57902025-04-30T06:30:03.808ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
57912025-04-30T06:30:03.808ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
57922025-04-30T06:30:03.808ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
57932025-04-30T06:30:03.808ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57942025-04-30T06:30:03.834ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57952025-04-30T06:30:03.834ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57962025-04-30T06:30:03.834ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
57972025-04-30T06:30:03.836ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57982025-04-30T06:30:03.836ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
57992025-04-30T06:30:03.836ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
58002025-04-30T06:30:03.842ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
58012025-04-30T06:30:03.842ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58022025-04-30T06:30:03.843ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6c52d62-f221-4c89-9282-e4a5980b188d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
58032025-04-30T06:30:03.844ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
58042025-04-30T06:30:03.844ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58052025-04-30T06:30:03.844ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
58062025-04-30T06:30:03.845ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58072025-04-30T06:30:03.850ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e0a9311-6ce0-4aa6-b1ac-e4c617f09723", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
58082025-04-30T06:30:03.851ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
58092025-04-30T06:30:03.856ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9972509-49e4-4fca-a16c-b0bb23bec83e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:02 GMT"} })
58102025-04-30T06:30:03.858ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
58112025-04-30T06:30:03.867ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7ff7fbf-dd69-4969-b09c-20e4d5f9770c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58122025-04-30T06:30:03.867ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
58132025-04-30T06:30:03.874ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c0d0ac4-2a11-45ec-9df7-2ea60b32ec6e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58142025-04-30T06:30:03.874ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
58152025-04-30T06:30:03.881ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66cfed07-7b8e-4f1c-b523-3591000f4de7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58162025-04-30T06:30:03.882ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
58172025-04-30T06:30:03.887ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
58182025-04-30T06:30:03.891ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "167a075c-f176-4018-ad51-70872dccb688", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58192025-04-30T06:30:03.892ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
58202025-04-30T06:30:03.898ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
58212025-04-30T06:30:03.901ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "206c04f9-a155-429d-ace7-3ae601e347d9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58222025-04-30T06:30:03.901ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
58232025-04-30T06:30:03.903ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd0ef1f8-a3e9-4ed7-b1f8-a623b9fb4bfb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58242025-04-30T06:30:03.903ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
58252025-04-30T06:30:03.905ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73c11472-f418-493e-a172-65f9656a73ea", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58262025-04-30T06:30:03.905ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
58272025-04-30T06:30:03.907ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5f8791c-2001-4f53-ba35-aa7d4ce55458", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58282025-04-30T06:30:03.908ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
58292025-04-30T06:30:03.910ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bb2d300-0f65-4b39-bb2c-aa5413f7bc39", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58302025-04-30T06:30:03.910ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
58312025-04-30T06:30:03.915ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d0f4cd7-567a-4c32-a256-43c744f5ee60", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58322025-04-30T06:30:03.916ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
58332025-04-30T06:30:03.919ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f583a5fe-afd4-494a-8081-b9c390e4924f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58342025-04-30T06:30:03.920ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
58352025-04-30T06:30:03.922ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d40b7a1-776f-4686-980c-93bf97a4b79b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58362025-04-30T06:30:03.923ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
58372025-04-30T06:30:03.927ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5474ca8-883c-49ee-827a-bbe33686a856", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58382025-04-30T06:30:03.927ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
58392025-04-30T06:30:03.932ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8b59f8b-1f03-4dd1-a2f0-dc6b962b2856", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58402025-04-30T06:30:03.933ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
58412025-04-30T06:30:03.936ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58a4b25f-c6fe-4ec0-b38d-bd2fa5cd43f5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58422025-04-30T06:30:03.936ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
58432025-04-30T06:30:03.937ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58442025-04-30T06:30:03.938ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58452025-04-30T06:30:03.938ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58462025-04-30T06:30:03.938ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
58472025-04-30T06:30:03.939ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58482025-04-30T06:30:03.944ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58492025-04-30T06:30:03.955ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58502025-04-30T06:30:03.955ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58512025-04-30T06:30:03.955ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
58522025-04-30T06:30:03.955ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58532025-04-30T06:30:03.960ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bca147c-55a6-4ce0-8391-cef5798805b5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58542025-04-30T06:30:03.961ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
58552025-04-30T06:30:03.962ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58562025-04-30T06:30:03.962ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
58572025-04-30T06:30:03.963ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58582025-04-30T06:30:03.973ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bbb7477-da98-458a-9650-bd30329bc728", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58592025-04-30T06:30:03.973ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
58602025-04-30T06:30:03.999ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e48e6358-460c-40cf-9e3e-dda9c1b4788e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58612025-04-30T06:30:03.999ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
58622025-04-30T06:30:04.013ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12185ce2-ac92-4dc2-8b65-ac36c14101c7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58632025-04-30T06:30:04.014ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
58642025-04-30T06:30:04.032ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f783d91-6039-43ad-8e91-669305694c91", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58652025-04-30T06:30:04.032ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
58662025-04-30T06:30:04.035ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2253258d-93a6-42c5-bddb-a78bed95a7ef", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58672025-04-30T06:30:04.036ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
58682025-04-30T06:30:04.040ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ba0e909-89b8-403c-935d-850d3fc914e7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58692025-04-30T06:30:04.040ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
58702025-04-30T06:30:04.045ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76f9d359-6328-4235-b306-0a6289e296c5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58712025-04-30T06:30:04.045ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
58722025-04-30T06:30:04.045ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
58732025-04-30T06:30:04.048ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "73a8730b-a938-4b9c-ba14-a8672d534460", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58742025-04-30T06:30:04.049ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
58752025-04-30T06:30:04.051ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31c096ca-9729-442d-8a1f-2c27cb8f9add", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58762025-04-30T06:30:04.051ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
58772025-04-30T06:30:04.053ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12ef2885-d6e3-4baa-ba45-512faa8e139e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58782025-04-30T06:30:04.053ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
58792025-04-30T06:30:04.055ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "565987fc-ae20-463b-8298-d3eed40a097c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58802025-04-30T06:30:04.055ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
58812025-04-30T06:30:04.056ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93f82b6c-29e1-4b87-8291-c00847adf331", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58822025-04-30T06:30:04.056ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
58832025-04-30T06:30:04.057ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ff71970-38a9-41af-891f-610c442c1fce", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58842025-04-30T06:30:04.057ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
58852025-04-30T06:30:04.058ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7455165a-14f3-41cd-bbbe-e3bb3e22446f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58862025-04-30T06:30:04.058ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
58872025-04-30T06:30:04.059ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20b29f14-6c2e-4182-8bbc-6d22fe847960", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58882025-04-30T06:30:04.060ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
58892025-04-30T06:30:04.061ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e7610ad-abd4-4a17-b87d-030c6cea2b2b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58902025-04-30T06:30:04.061ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
58912025-04-30T06:30:04.063ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cfb9f90-c065-442f-8d2d-6a8ab559a14f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58922025-04-30T06:30:04.063ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
58932025-04-30T06:30:04.066ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb0160d0-2826-4c45-942d-947ddd4585d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58942025-04-30T06:30:04.066ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
58952025-04-30T06:30:04.068ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70873893-d7b8-4927-b1d5-ab528ea37c97", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58962025-04-30T06:30:04.068ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
58972025-04-30T06:30:04.070ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b243c5b-32e5-4a89-9dae-ba2e51bcaf41", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
58982025-04-30T06:30:04.071ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
58992025-04-30T06:30:04.073ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13d97be2-afb5-49a6-8c4c-8e428a573e31", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
59002025-04-30T06:30:04.073ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
59012025-04-30T06:30:04.075ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1021fabb-f341-4664-95b1-5ff9d2a2efa9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
59022025-04-30T06:30:04.089ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
59032025-04-30T06:30:04.094ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab4d98d4-e735-4d2a-ab43-1e005febdc89", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
59042025-04-30T06:30:04.095ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
59052025-04-30T06:30:04.096ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6a287d9-36b6-4af7-bf90-6d99abdb1e88", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
59062025-04-30T06:30:04.096ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
59072025-04-30T06:30:04.097ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59082025-04-30T06:30:04.097ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59092025-04-30T06:30:04.097ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59102025-04-30T06:30:04.097ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
59112025-04-30T06:30:04.098ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59122025-04-30T06:30:04.099ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55e2dd51-70ae-46af-ab12-b8b861e7a953", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
59132025-04-30T06:30:04.100ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
59142025-04-30T06:30:04.103ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59152025-04-30T06:30:04.110ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59162025-04-30T06:30:04.110ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59172025-04-30T06:30:04.110ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
59182025-04-30T06:30:04.111ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59192025-04-30T06:30:04.111ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91aa8a05-a029-4a11-b038-1892defeda50", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
59202025-04-30T06:30:04.111ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59212025-04-30T06:30:04.111ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59222025-04-30T06:30:04.111ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59232025-04-30T06:30:04.111ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59242025-04-30T06:30:04.112ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59252025-04-30T06:30:04.116ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59262025-04-30T06:30:04.116ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
59272025-04-30T06:30:04.117ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59282025-04-30T06:30:04.117ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59292025-04-30T06:30:04.123ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59302025-04-30T06:30:04.123ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59312025-04-30T06:30:04.123ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59322025-04-30T06:30:04.123ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59332025-04-30T06:30:04.123ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59342025-04-30T06:30:04.124ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59352025-04-30T06:30:04.127ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59362025-04-30T06:30:04.132ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59372025-04-30T06:30:04.132ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59382025-04-30T06:30:04.132ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59392025-04-30T06:30:04.133ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59402025-04-30T06:30:04.224ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
59412025-04-30T06:30:04.243ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
59422025-04-30T06:30:04.245ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:53991 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
59432025-04-30T06:30:04.245ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:53991 req_id = b5988000-9c51-48ef-8460-284e83819b2d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
59442025-04-30T06:30:04.246ZINFOtest_image_promotion (dropshot): request completed latency_us = 789 local_addr = [::1]:60070 method = GET remote_addr = [::1]:53991 req_id = b5988000-9c51-48ef-8460-284e83819b2d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
59452025-04-30T06:30:04.249ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5988000-9c51-48ef-8460-284e83819b2d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
59462025-04-30T06:30:04.252ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
59472025-04-30T06:30:04.252ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
59482025-04-30T06:30:04.253ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
59492025-04-30T06:30:04.255ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
59502025-04-30T06:30:04.255ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59512025-04-30T06:30:04.256ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59522025-04-30T06:30:04.256ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59532025-04-30T06:30:04.257ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59542025-04-30T06:30:04.260ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59552025-04-30T06:30:04.265ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
59562025-04-30T06:30:04.267ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59572025-04-30T06:30:04.267ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59582025-04-30T06:30:04.268ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59592025-04-30T06:30:04.292ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
59602025-04-30T06:30:04.309ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
59612025-04-30T06:30:04.310ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
59622025-04-30T06:30:04.310ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 591.62764ms iteration = 16 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
59632025-04-30T06:30:04.310ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 17 reason = Signaled
59642025-04-30T06:30:04.310ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
59652025-04-30T06:30:04.310ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59662025-04-30T06:30:04.310ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59672025-04-30T06:30:04.310ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59682025-04-30T06:30:04.311ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59692025-04-30T06:30:04.314ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59702025-04-30T06:30:04.328ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
59712025-04-30T06:30:04.328ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59722025-04-30T06:30:04.328ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59732025-04-30T06:30:04.328ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59742025-04-30T06:30:04.333ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
59752025-04-30T06:30:04.333ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
59762025-04-30T06:30:04.334ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5d 0e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ]............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52644 req_id = 452668b3-8b79-45bb-94b0-e9edfb1d112c
59772025-04-30T06:30:04.334ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23822,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52644 req_id = 452668b3-8b79-45bb-94b0-e9edfb1d112c
59782025-04-30T06:30:04.334ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
59792025-04-30T06:30:04.334ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
59802025-04-30T06:30:04.334ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52644 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 452668b3-8b79-45bb-94b0-e9edfb1d112c
59812025-04-30T06:30:04.335ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3469, tv_nsec: 134326985 } })
59822025-04-30T06:30:04.335ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
59832025-04-30T06:30:04.335ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f6 2c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .,..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60278 req_id = f7950834-83ee-4dbf-a840-70674e762a6a
59842025-04-30T06:30:04.335ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63020,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60278 req_id = f7950834-83ee-4dbf-a840-70674e762a6a
59852025-04-30T06:30:04.336ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
59862025-04-30T06:30:04.336ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60278 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f7950834-83ee-4dbf-a840-70674e762a6a
59872025-04-30T06:30:04.412ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
59882025-04-30T06:30:04.412ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59892025-04-30T06:30:04.413ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59902025-04-30T06:30:04.413ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59912025-04-30T06:30:04.413ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
59922025-04-30T06:30:04.428ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59932025-04-30T06:30:04.432ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6b e9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c k............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62558 req_id = 33041586-b06c-46e0-afac-49fd6d0bd0b0
59942025-04-30T06:30:04.432ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27625,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62558 req_id = 33041586-b06c-46e0-afac-49fd6d0bd0b0
59952025-04-30T06:30:04.433ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
59962025-04-30T06:30:04.433ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27625, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62558 req_id = 33041586-b06c-46e0-afac-49fd6d0bd0b0
59972025-04-30T06:30:04.433ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
59982025-04-30T06:30:04.433ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
59992025-04-30T06:30:04.433ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
60002025-04-30T06:30:04.448ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60012025-04-30T06:30:04.456ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60022025-04-30T06:30:04.456ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60032025-04-30T06:30:04.456ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
60042025-04-30T06:30:04.458ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60052025-04-30T06:30:04.461ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5295b8ac-d09f-4212-aca8-092576775532", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60062025-04-30T06:30:04.461ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
60072025-04-30T06:30:04.462ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60082025-04-30T06:30:04.463ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
60092025-04-30T06:30:04.464ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60102025-04-30T06:30:04.466ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bff814da-1950-4577-ae01-e8e61a919b8e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60112025-04-30T06:30:04.468ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
60122025-04-30T06:30:04.473ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd2c11ac-6899-48c4-b491-0b7172ed5e91", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60132025-04-30T06:30:04.473ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
60142025-04-30T06:30:04.476ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e9d670d-5ebc-4551-a762-eafe500f1a9b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60152025-04-30T06:30:04.477ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
60162025-04-30T06:30:04.481ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a59da8e2-951c-42f3-8b5a-a279eb8fc842", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60172025-04-30T06:30:04.481ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
60182025-04-30T06:30:04.486ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3a6605a-9b01-4f24-a5cd-cec5ccebbe6d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60192025-04-30T06:30:04.486ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
60202025-04-30T06:30:04.490ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fed46774-40ce-4e4d-bfbd-d20ae56e66eb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60212025-04-30T06:30:04.491ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
60222025-04-30T06:30:04.496ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66a50acb-0a11-445e-aeae-3ea61b5aa27c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60232025-04-30T06:30:04.496ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
60242025-04-30T06:30:04.496ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
60252025-04-30T06:30:04.498ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07252371-ba24-4c92-b968-779e270f8bb1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60262025-04-30T06:30:04.498ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
60272025-04-30T06:30:04.499ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e4e659a-7482-4fd7-a26e-9d5f2876f30e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60282025-04-30T06:30:04.499ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
60292025-04-30T06:30:04.501ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdb3a92d-5455-4933-bf71-3a7d82ff20d5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60302025-04-30T06:30:04.501ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
60312025-04-30T06:30:04.502ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "99f1c274-01bf-42bd-b06e-535441e7a29f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60322025-04-30T06:30:04.502ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
60332025-04-30T06:30:04.504ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07afbdfa-a75a-495e-9c14-16e1424a3f0f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60342025-04-30T06:30:04.504ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
60352025-04-30T06:30:04.505ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
60362025-04-30T06:30:04.505ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60372025-04-30T06:30:04.506ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 63cf0942-28e2-47f5-98cf-2e5030fda7b8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60382025-04-30T06:30:04.507ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 63cf0942-28e2-47f5-98cf-2e5030fda7b8 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60392025-04-30T06:30:04.507ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 63cf0942-28e2-47f5-98cf-2e5030fda7b8 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60402025-04-30T06:30:04.508ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 63cf0942-28e2-47f5-98cf-2e5030fda7b8 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60412025-04-30T06:30:04.510ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff9024bc-61dd-47ed-86fe-acf981dab3d6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60422025-04-30T06:30:04.510ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
60432025-04-30T06:30:04.511ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 5237 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 63cf0942-28e2-47f5-98cf-2e5030fda7b8 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60442025-04-30T06:30:04.513ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "63cf0942-28e2-47f5-98cf-2e5030fda7b8", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60452025-04-30T06:30:04.513ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
60462025-04-30T06:30:04.513ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 2a414484-4e8b-4747-ba38-bed01373edae uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
60472025-04-30T06:30:04.514ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 2a414484-4e8b-4747-ba38-bed01373edae resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
60482025-04-30T06:30:04.514ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 2a414484-4e8b-4747-ba38-bed01373edae roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
60492025-04-30T06:30:04.515ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 2a414484-4e8b-4747-ba38-bed01373edae resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
60502025-04-30T06:30:04.517ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37e219b1-3ca1-4d0f-999f-4aa0b8e28ec1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60512025-04-30T06:30:04.517ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
60522025-04-30T06:30:04.518ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 4142 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 2a414484-4e8b-4747-ba38-bed01373edae response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
60532025-04-30T06:30:04.518ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a414484-4e8b-4747-ba38-bed01373edae", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60542025-04-30T06:30:04.518ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60552025-04-30T06:30:04.518ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
60562025-04-30T06:30:04.518ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
60572025-04-30T06:30:04.521ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6b2238c-1a30-4ccf-8805-5381e072b838", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60582025-04-30T06:30:04.521ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
60592025-04-30T06:30:04.524ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d9d288a-a1da-4d4c-9aa6-0043159f6c4b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60602025-04-30T06:30:04.524ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
60612025-04-30T06:30:04.528ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae36f251-8b86-49d8-b06c-312c74aab113", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60622025-04-30T06:30:04.528ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
60632025-04-30T06:30:04.530ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69de7ff7-df02-42c5-8eda-a1cbd586b406", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60642025-04-30T06:30:04.530ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
60652025-04-30T06:30:04.532ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "589b3edd-77ff-482e-9fa2-71a3dfc0b144", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60662025-04-30T06:30:04.532ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
60672025-04-30T06:30:04.533ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6da03dd-c2c7-40a8-add8-a9b277f44a3c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60682025-04-30T06:30:04.533ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
60692025-04-30T06:30:04.535ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2049e14-b2b5-4302-a5bc-80442644280d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60702025-04-30T06:30:04.535ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
60712025-04-30T06:30:04.538ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60722025-04-30T06:30:04.540ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f25bfffb-f21c-4bd4-9545-80c6d8826d2a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60732025-04-30T06:30:04.540ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
60742025-04-30T06:30:04.540ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60752025-04-30T06:30:04.545ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60762025-04-30T06:30:04.547ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60772025-04-30T06:30:04.547ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60782025-04-30T06:30:04.547ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60792025-04-30T06:30:04.548ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60802025-04-30T06:30:04.548ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60812025-04-30T06:30:04.548ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60822025-04-30T06:30:04.548ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
60832025-04-30T06:30:04.549ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60842025-04-30T06:30:04.551ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a312931-0f6c-4889-acd3-908dad554a90", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60852025-04-30T06:30:04.551ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
60862025-04-30T06:30:04.552ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60872025-04-30T06:30:04.559ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60882025-04-30T06:30:04.559ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60892025-04-30T06:30:04.559ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
60902025-04-30T06:30:04.560ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60912025-04-30T06:30:04.562ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60922025-04-30T06:30:04.562ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
60932025-04-30T06:30:04.563ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60942025-04-30T06:30:04.566ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef662155-89c6-426b-9e61-6fb2808d4782", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60952025-04-30T06:30:04.566ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
60962025-04-30T06:30:04.569ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f0d4874-cc8a-4226-8493-b0e0025fd463", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60972025-04-30T06:30:04.569ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
60982025-04-30T06:30:04.572ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b7b550e-3af3-474c-a6e6-cdf4dd9decad", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
60992025-04-30T06:30:04.573ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
61002025-04-30T06:30:04.577ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94e6708f-f9bf-47c4-84c1-b110007d6211", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61012025-04-30T06:30:04.577ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
61022025-04-30T06:30:04.589ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0968bfd-7779-49da-9d63-4a4ebcc1d224", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61032025-04-30T06:30:04.589ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
61042025-04-30T06:30:04.593ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "555446f8-42b9-426b-9a29-179bd45585a6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61052025-04-30T06:30:04.593ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
61062025-04-30T06:30:04.596ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09b37f1b-648b-4aa2-9e56-fe4e80293979", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61072025-04-30T06:30:04.596ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
61082025-04-30T06:30:04.598ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6edbb5be-ed5a-405d-8e2c-915592722d5a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61092025-04-30T06:30:04.598ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
61102025-04-30T06:30:04.600ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "512118e8-6add-4cb1-b75c-5166c55d6fcd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61112025-04-30T06:30:04.600ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
61122025-04-30T06:30:04.601ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
61132025-04-30T06:30:04.602ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0bb076c-c36f-48c2-8440-754beade0d56", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61142025-04-30T06:30:04.602ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
61152025-04-30T06:30:04.603ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bd53d3f-2aea-4a06-8d4d-126a73107f75", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61162025-04-30T06:30:04.603ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
61172025-04-30T06:30:04.605ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70c79bab-45c4-46f2-977a-d4226548b55e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61182025-04-30T06:30:04.605ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
61192025-04-30T06:30:04.608ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa3be493-4c51-4344-a3c7-d48c5949c961", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61202025-04-30T06:30:04.608ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
61212025-04-30T06:30:04.610ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ab35c75-6097-4842-8859-49619445e695", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61222025-04-30T06:30:04.610ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
61232025-04-30T06:30:04.612ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d43c4b8-3f11-4974-85e7-d5503e8802d6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61242025-04-30T06:30:04.612ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
61252025-04-30T06:30:04.615ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b166873e-bb6e-40a7-8f52-3700567e7cc8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61262025-04-30T06:30:04.615ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
61272025-04-30T06:30:04.617ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbe34b44-aaca-4969-96a8-44b6f2b3937b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61282025-04-30T06:30:04.617ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
61292025-04-30T06:30:04.618ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e49d2616-7712-46ed-86d7-ed0bb9a5ad25", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61302025-04-30T06:30:04.618ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
61312025-04-30T06:30:04.618ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5d40575-dd87-45f4-ac9b-7c29f327b51d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61322025-04-30T06:30:04.619ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
61332025-04-30T06:30:04.620ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "830e259a-7601-482d-9c37-806f544239e4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61342025-04-30T06:30:04.620ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
61352025-04-30T06:30:04.621ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b5b0c84-84ac-4318-bc94-37fd442081bf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61362025-04-30T06:30:04.621ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61372025-04-30T06:30:04.621ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61382025-04-30T06:30:04.621ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61392025-04-30T06:30:04.621ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61402025-04-30T06:30:04.621ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61412025-04-30T06:30:04.624ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61422025-04-30T06:30:04.635ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61432025-04-30T06:30:04.635ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61442025-04-30T06:30:04.635ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61452025-04-30T06:30:04.635ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61462025-04-30T06:30:04.635ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61472025-04-30T06:30:04.636ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61482025-04-30T06:30:04.639ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61492025-04-30T06:30:04.645ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61502025-04-30T06:30:04.645ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61512025-04-30T06:30:04.645ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61522025-04-30T06:30:04.645ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61532025-04-30T06:30:04.702ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
61542025-04-30T06:30:04.702ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
61552025-04-30T06:30:04.702ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61562025-04-30T06:30:04.702ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61572025-04-30T06:30:04.703ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61582025-04-30T06:30:04.703ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
61592025-04-30T06:30:04.703ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61602025-04-30T06:30:04.703ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:35570 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
61612025-04-30T06:30:04.704ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:35570 req_id = ff0f3b26-278c-4c7d-80a4-d7c123dab6d7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
61622025-04-30T06:30:04.705ZINFOtest_image_promotion (dropshot): request completed latency_us = 445 local_addr = [::1]:60070 method = GET remote_addr = [::1]:35570 req_id = ff0f3b26-278c-4c7d-80a4-d7c123dab6d7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
61632025-04-30T06:30:04.705ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff0f3b26-278c-4c7d-80a4-d7c123dab6d7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:03 GMT"} })
61642025-04-30T06:30:04.707ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
61652025-04-30T06:30:04.707ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
61662025-04-30T06:30:04.707ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
61672025-04-30T06:30:04.708ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
61682025-04-30T06:30:04.708ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61692025-04-30T06:30:04.708ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61702025-04-30T06:30:04.708ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61712025-04-30T06:30:04.709ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61722025-04-30T06:30:04.709ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61732025-04-30T06:30:04.716ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
61742025-04-30T06:30:04.716ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61752025-04-30T06:30:04.716ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
61762025-04-30T06:30:04.717ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61772025-04-30T06:30:04.719ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61782025-04-30T06:30:04.723ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
61792025-04-30T06:30:04.723ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61802025-04-30T06:30:04.723ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61812025-04-30T06:30:04.724ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61822025-04-30T06:30:04.724ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61832025-04-30T06:30:04.724ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
61842025-04-30T06:30:04.725ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61852025-04-30T06:30:04.750ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
61862025-04-30T06:30:04.753ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
61872025-04-30T06:30:04.753ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
61882025-04-30T06:30:04.754ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
61892025-04-30T06:30:04.760ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
61902025-04-30T06:30:04.760ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
61912025-04-30T06:30:04.764ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
61922025-04-30T06:30:04.764ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
61932025-04-30T06:30:04.764ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 453.828043ms iteration = 17 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
61942025-04-30T06:30:04.764ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 18 reason = Signaled
61952025-04-30T06:30:04.764ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
61962025-04-30T06:30:04.764ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61972025-04-30T06:30:04.764ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61982025-04-30T06:30:04.764ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61992025-04-30T06:30:04.765ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62002025-04-30T06:30:04.771ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62012025-04-30T06:30:04.775ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62022025-04-30T06:30:04.775ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62032025-04-30T06:30:04.775ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62042025-04-30T06:30:04.778ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62052025-04-30T06:30:04.779ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
62062025-04-30T06:30:04.779ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
62072025-04-30T06:30:04.779ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62082025-04-30T06:30:04.782ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
62092025-04-30T06:30:04.782ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
62102025-04-30T06:30:04.783ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6a 77 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 jw..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57559 req_id = a437d13c-8927-4810-9064-b6928dde676b
62112025-04-30T06:30:04.783ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27255,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57559 req_id = a437d13c-8927-4810-9064-b6928dde676b
62122025-04-30T06:30:04.783ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
62132025-04-30T06:30:04.784ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62142025-04-30T06:30:04.784ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57559 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a437d13c-8927-4810-9064-b6928dde676b
62152025-04-30T06:30:04.784ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3469, tv_nsec: 582710329 } })
62162025-04-30T06:30:04.784ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
62172025-04-30T06:30:04.784ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 15 ce 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51533 req_id = 79d39154-9b8a-45aa-8a2a-01ba6f3d0a92
62182025-04-30T06:30:04.784ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5582,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51533 req_id = 79d39154-9b8a-45aa-8a2a-01ba6f3d0a92
62192025-04-30T06:30:04.784ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62202025-04-30T06:30:04.784ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51533 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 79d39154-9b8a-45aa-8a2a-01ba6f3d0a92
62212025-04-30T06:30:04.846ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
62222025-04-30T06:30:04.846ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62232025-04-30T06:30:04.846ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62242025-04-30T06:30:04.846ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62252025-04-30T06:30:04.846ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
62262025-04-30T06:30:04.847ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62272025-04-30T06:30:04.849ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 61 18 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c a............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61901 req_id = 3a7bca2d-6e3d-49e6-8218-d2a83ee8012d
62282025-04-30T06:30:04.849ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24856,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61901 req_id = 3a7bca2d-6e3d-49e6-8218-d2a83ee8012d
62292025-04-30T06:30:04.849ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
62302025-04-30T06:30:04.849ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24856, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61901 req_id = 3a7bca2d-6e3d-49e6-8218-d2a83ee8012d
62312025-04-30T06:30:04.851ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
62322025-04-30T06:30:04.851ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
62332025-04-30T06:30:04.851ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
62342025-04-30T06:30:04.852ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62352025-04-30T06:30:04.859ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62362025-04-30T06:30:04.859ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62372025-04-30T06:30:04.859ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
62382025-04-30T06:30:04.859ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62392025-04-30T06:30:04.861ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
62402025-04-30T06:30:04.864ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62412025-04-30T06:30:04.864ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
62422025-04-30T06:30:04.864ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62432025-04-30T06:30:04.864ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "326a3e2f-8b46-48a7-a009-40dd417e4308", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62442025-04-30T06:30:04.864ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
62452025-04-30T06:30:04.867ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8390626-8229-4892-aa3a-bc3008e66bfb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62462025-04-30T06:30:04.867ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
62472025-04-30T06:30:04.873ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0030983d-aded-47d1-86ca-785c650ca065", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62482025-04-30T06:30:04.873ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
62492025-04-30T06:30:04.880ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97418cc8-7a4b-4de9-9383-2a974ddf1b2e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62502025-04-30T06:30:04.880ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
62512025-04-30T06:30:04.884ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcca47bd-401c-4890-8b4b-5c478d40e9cd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62522025-04-30T06:30:04.884ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
62532025-04-30T06:30:04.892ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bdf188d-5e36-4b74-918a-7718e59d0380", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62542025-04-30T06:30:04.893ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
62552025-04-30T06:30:04.897ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a4b6b66-0e3a-4e35-b2f9-7c6b87149907", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62562025-04-30T06:30:04.897ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
62572025-04-30T06:30:04.901ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
62582025-04-30T06:30:04.903ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2555516c-7911-4c5b-84d6-2c4b75ed4fea", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62592025-04-30T06:30:04.903ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
62602025-04-30T06:30:04.905ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e185bc1c-bac7-43f3-85e5-d34e47739c46", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62612025-04-30T06:30:04.905ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
62622025-04-30T06:30:04.906ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "977cccd9-997c-436e-832d-ea30c9d2dc1d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62632025-04-30T06:30:04.907ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
62642025-04-30T06:30:04.908ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c42946f-4631-4022-824d-35a6601f8f68", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62652025-04-30T06:30:04.908ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
62662025-04-30T06:30:04.911ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40f29ae3-aa6e-4af2-9d7f-8953127d60ed", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62672025-04-30T06:30:04.911ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
62682025-04-30T06:30:04.913ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1e5ceaf-d46a-43d0-b67f-b73c8c8fc1f3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62692025-04-30T06:30:04.913ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
62702025-04-30T06:30:04.917ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49f292ab-e64b-4486-b979-85b57a947e68", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62712025-04-30T06:30:04.917ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
62722025-04-30T06:30:04.931ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e6de460-d953-4657-8b13-b156b0da2516", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62732025-04-30T06:30:04.931ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
62742025-04-30T06:30:04.935ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34a129de-f979-40d8-911f-ff38d133e253", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62752025-04-30T06:30:04.935ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
62762025-04-30T06:30:04.945ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0fb2e6e-af9a-4fc7-b1db-7da253872d71", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62772025-04-30T06:30:04.945ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
62782025-04-30T06:30:04.949ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82dee7a0-500c-4d53-8e94-6cc95aa6fe02", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62792025-04-30T06:30:04.949ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
62802025-04-30T06:30:04.953ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62812025-04-30T06:30:04.954ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62822025-04-30T06:30:04.954ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62832025-04-30T06:30:04.954ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
62842025-04-30T06:30:04.955ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62852025-04-30T06:30:04.959ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62862025-04-30T06:30:04.966ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62872025-04-30T06:30:04.966ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62882025-04-30T06:30:04.966ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
62892025-04-30T06:30:04.966ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62902025-04-30T06:30:04.977ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62912025-04-30T06:30:04.978ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
62922025-04-30T06:30:04.978ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62932025-04-30T06:30:05.006ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3763322c-03f4-4bd6-962a-a9a8d3cbf744", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62942025-04-30T06:30:05.014ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
62952025-04-30T06:30:05.022ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88a184e5-4db1-4ae3-b2b7-9751298f25bc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62962025-04-30T06:30:05.022ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
62972025-04-30T06:30:05.024ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "505de75c-b6bb-4cab-a10f-115314e15eb5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
62982025-04-30T06:30:05.025ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
62992025-04-30T06:30:05.028ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab7a7999-f8bd-4f8d-a9c7-b769e2fef877", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63002025-04-30T06:30:05.028ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
63012025-04-30T06:30:05.032ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59d53c09-5e1a-4ddc-8a0c-5d18c730df5e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63022025-04-30T06:30:05.032ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
63032025-04-30T06:30:05.035ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8448bdda-9794-4d3d-b1a0-9640bcb35999", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63042025-04-30T06:30:05.035ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
63052025-04-30T06:30:05.039ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2848052-b7ec-4b79-8723-c08819da3a98", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63062025-04-30T06:30:05.039ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
63072025-04-30T06:30:05.046ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d270df54-ee24-4d98-84f4-8fae719c049d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63082025-04-30T06:30:05.047ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
63092025-04-30T06:30:05.049ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
63102025-04-30T06:30:05.053ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88eef838-1989-4d5e-95c6-612069b03933", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63112025-04-30T06:30:05.054ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
63122025-04-30T06:30:05.055ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8ecee6b-c3d3-4439-bf35-7b8d8ddff944", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63132025-04-30T06:30:05.055ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
63142025-04-30T06:30:05.057ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8b593a5-9f0c-4534-b01c-2370968d574e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63152025-04-30T06:30:05.057ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
63162025-04-30T06:30:05.062ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdea1e4d-8483-4898-a15c-29f9a06b76e5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63172025-04-30T06:30:05.063ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
63182025-04-30T06:30:05.065ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23a7bf0a-71ce-453d-9f5b-34aa27cbe74f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63192025-04-30T06:30:05.066ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
63202025-04-30T06:30:05.067ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e797999-868c-46d1-be1a-fd861b6c608d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63212025-04-30T06:30:05.067ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
63222025-04-30T06:30:05.068ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7200a4e6-2845-4f71-9d0b-dd607977dca3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63232025-04-30T06:30:05.068ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
63242025-04-30T06:30:05.069ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1bc3848-230d-4cb6-bfc0-f6d4cce30beb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63252025-04-30T06:30:05.070ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
63262025-04-30T06:30:05.074ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c26aed2-7938-482b-a302-8cb92f3cd4fa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63272025-04-30T06:30:05.074ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
63282025-04-30T06:30:05.079ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "263c8076-416c-4b2a-8b10-c4e3d1b4570b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63292025-04-30T06:30:05.079ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
63302025-04-30T06:30:05.081ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea4a697f-ff3d-434a-b0fa-9a71b53629b5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63312025-04-30T06:30:05.081ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
63322025-04-30T06:30:05.083ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "101b090b-daa8-4c12-a252-1b63f49a8144", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63332025-04-30T06:30:05.084ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
63342025-04-30T06:30:05.087ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5512dbf9-e2fc-4218-b90e-789d637636b6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63352025-04-30T06:30:05.087ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
63362025-04-30T06:30:05.089ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9295b2e-74e5-4daa-a788-c13e3354c1fa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63372025-04-30T06:30:05.089ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
63382025-04-30T06:30:05.091ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c92879a9-0bb9-44e6-a173-f098d061f6c8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63392025-04-30T06:30:05.092ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
63402025-04-30T06:30:05.092ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5483ba4a-91ca-4f23-b89b-2c258602f7a8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63412025-04-30T06:30:05.093ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
63422025-04-30T06:30:05.093ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfe06292-b6fd-4d60-9a02-5f6503f66375", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63432025-04-30T06:30:05.094ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
63442025-04-30T06:30:05.095ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba478ae8-20a8-45a1-b1fb-89e07365421b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63452025-04-30T06:30:05.095ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
63462025-04-30T06:30:05.097ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e4f3ac7-f05d-4057-abc4-ba843712e314", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63472025-04-30T06:30:05.097ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63482025-04-30T06:30:05.097ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63492025-04-30T06:30:05.097ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63502025-04-30T06:30:05.097ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63512025-04-30T06:30:05.098ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63522025-04-30T06:30:05.100ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63532025-04-30T06:30:05.100ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63542025-04-30T06:30:05.100ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63552025-04-30T06:30:05.100ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
63562025-04-30T06:30:05.102ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63572025-04-30T06:30:05.103ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63582025-04-30T06:30:05.109ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
63592025-04-30T06:30:05.109ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63602025-04-30T06:30:05.109ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63612025-04-30T06:30:05.109ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63622025-04-30T06:30:05.109ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63632025-04-30T06:30:05.109ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63642025-04-30T06:30:05.111ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63652025-04-30T06:30:05.119ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63662025-04-30T06:30:05.119ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63672025-04-30T06:30:05.119ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
63682025-04-30T06:30:05.120ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63692025-04-30T06:30:05.122ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63702025-04-30T06:30:05.128ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
63712025-04-30T06:30:05.129ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63722025-04-30T06:30:05.129ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63732025-04-30T06:30:05.129ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63742025-04-30T06:30:05.131ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63752025-04-30T06:30:05.131ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
63762025-04-30T06:30:05.132ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63772025-04-30T06:30:05.194ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
63782025-04-30T06:30:05.194ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
63792025-04-30T06:30:05.194ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:37226 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
63802025-04-30T06:30:05.195ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:37226 req_id = d9db2874-b4a0-4740-b7f4-f1a32db04f80 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
63812025-04-30T06:30:05.195ZINFOtest_image_promotion (dropshot): request completed latency_us = 448 local_addr = [::1]:60070 method = GET remote_addr = [::1]:37226 req_id = d9db2874-b4a0-4740-b7f4-f1a32db04f80 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
63822025-04-30T06:30:05.196ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9db2874-b4a0-4740-b7f4-f1a32db04f80", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
63832025-04-30T06:30:05.201ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
63842025-04-30T06:30:05.201ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
63852025-04-30T06:30:05.201ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
63862025-04-30T06:30:05.206ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
63872025-04-30T06:30:05.206ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63882025-04-30T06:30:05.206ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63892025-04-30T06:30:05.206ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63902025-04-30T06:30:05.207ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63912025-04-30T06:30:05.210ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63922025-04-30T06:30:05.216ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
63932025-04-30T06:30:05.216ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63942025-04-30T06:30:05.217ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63952025-04-30T06:30:05.217ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63962025-04-30T06:30:05.247ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
63972025-04-30T06:30:05.253ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
63982025-04-30T06:30:05.255ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
63992025-04-30T06:30:05.255ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 490.367025ms iteration = 18 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
64002025-04-30T06:30:05.255ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 19 reason = Signaled
64012025-04-30T06:30:05.255ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64022025-04-30T06:30:05.255ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64032025-04-30T06:30:05.256ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64042025-04-30T06:30:05.256ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64052025-04-30T06:30:05.256ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64062025-04-30T06:30:05.259ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64072025-04-30T06:30:05.265ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64082025-04-30T06:30:05.265ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64092025-04-30T06:30:05.265ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64102025-04-30T06:30:05.266ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64112025-04-30T06:30:05.270ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
64122025-04-30T06:30:05.270ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
64132025-04-30T06:30:05.271ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 8c ce 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49721 req_id = 4769faa1-44f6-4d4a-a72a-1e40f8001f8b
64142025-04-30T06:30:05.271ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36046,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49721 req_id = 4769faa1-44f6-4d4a-a72a-1e40f8001f8b
64152025-04-30T06:30:05.271ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
64162025-04-30T06:30:05.271ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64172025-04-30T06:30:05.271ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49721 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4769faa1-44f6-4d4a-a72a-1e40f8001f8b
64182025-04-30T06:30:05.272ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3470, tv_nsec: 71171947 } })
64192025-04-30T06:30:05.272ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
64202025-04-30T06:30:05.272ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9e c8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55302 req_id = db09f2e8-3500-489d-8f75-d3e7163a4661
64212025-04-30T06:30:05.272ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40648,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55302 req_id = db09f2e8-3500-489d-8f75-d3e7163a4661
64222025-04-30T06:30:05.272ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64232025-04-30T06:30:05.273ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55302 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = db09f2e8-3500-489d-8f75-d3e7163a4661
64242025-04-30T06:30:05.344ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
64252025-04-30T06:30:05.346ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64262025-04-30T06:30:05.346ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64272025-04-30T06:30:05.346ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64282025-04-30T06:30:05.346ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
64292025-04-30T06:30:05.347ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64302025-04-30T06:30:05.348ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2a 61 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c *a..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63691 req_id = 80340e51-ea6a-4b44-bb8d-b6e616e5f91a
64312025-04-30T06:30:05.348ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10849,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63691 req_id = 80340e51-ea6a-4b44-bb8d-b6e616e5f91a
64322025-04-30T06:30:05.348ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
64332025-04-30T06:30:05.348ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10849, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63691 req_id = 80340e51-ea6a-4b44-bb8d-b6e616e5f91a
64342025-04-30T06:30:05.348ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
64352025-04-30T06:30:05.348ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
64362025-04-30T06:30:05.349ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
64372025-04-30T06:30:05.351ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64382025-04-30T06:30:05.364ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64392025-04-30T06:30:05.364ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64402025-04-30T06:30:05.365ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
64412025-04-30T06:30:05.365ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64422025-04-30T06:30:05.366ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "eff43e5b-3f88-4ba9-9140-99edf575b2a6", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64432025-04-30T06:30:05.367ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
64442025-04-30T06:30:05.371ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64452025-04-30T06:30:05.371ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
64462025-04-30T06:30:05.371ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64472025-04-30T06:30:05.383ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b75639b-2b1b-48f1-a62d-6784560db6cc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64482025-04-30T06:30:05.383ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
64492025-04-30T06:30:05.387ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01a93405-39d7-4bc1-942f-2ea15f51fb4f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64502025-04-30T06:30:05.387ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
64512025-04-30T06:30:05.392ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a6f2958-9e7c-480b-a300-e458d1c99bc0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64522025-04-30T06:30:05.392ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
64532025-04-30T06:30:05.396ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14172028-cecd-41e2-b749-ab04e0a517d8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64542025-04-30T06:30:05.396ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
64552025-04-30T06:30:05.402ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14f67bd0-ef36-4316-8f62-4319129f9004", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64562025-04-30T06:30:05.402ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
64572025-04-30T06:30:05.406ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6db0fb4-9be8-4283-b108-983ee04ef360", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64582025-04-30T06:30:05.406ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
64592025-04-30T06:30:05.411ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dc40f97-69e2-49b3-bdc6-ed0e301ee750", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64602025-04-30T06:30:05.412ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
64612025-04-30T06:30:05.416ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
64622025-04-30T06:30:05.419ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfa1d2cd-2033-41a5-ac2e-e1a440d66659", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64632025-04-30T06:30:05.419ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
64642025-04-30T06:30:05.421ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "acae3990-f46c-46f9-a1ba-491e1621c976", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64652025-04-30T06:30:05.421ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
64662025-04-30T06:30:05.422ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8761a0e-483b-4170-9c5e-7da20ed89a7a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64672025-04-30T06:30:05.422ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
64682025-04-30T06:30:05.423ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd9a292c-a50f-4b87-a8ff-9261ffc6f5d1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64692025-04-30T06:30:05.423ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
64702025-04-30T06:30:05.425ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "822ce604-c5b9-4e4a-a641-ff188b1d5450", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64712025-04-30T06:30:05.425ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
64722025-04-30T06:30:05.427ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "683b266a-edc1-402c-bd69-7d4ca66c46fb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64732025-04-30T06:30:05.427ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
64742025-04-30T06:30:05.429ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb10f4f8-2772-4695-9631-a446a8ffb310", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64752025-04-30T06:30:05.429ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
64762025-04-30T06:30:05.431ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd0cf6e8-699d-4e46-8fdf-266770125ee8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64772025-04-30T06:30:05.431ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
64782025-04-30T06:30:05.433ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08401fc7-25f1-4d37-a322-5c16b247e313", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64792025-04-30T06:30:05.433ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
64802025-04-30T06:30:05.435ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac348a13-49cd-4740-b80d-1c4a9daaaf3a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64812025-04-30T06:30:05.435ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
64822025-04-30T06:30:05.437ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "602f1b65-299b-4922-a9a0-0043b4c030e5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64832025-04-30T06:30:05.438ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
64842025-04-30T06:30:05.439ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dce23d6-1ac1-47d4-a2c4-d0ab4c7b3be0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64852025-04-30T06:30:05.439ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
64862025-04-30T06:30:05.440ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b046ed9f-deac-4767-aacb-0ef1fa02cf5b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64872025-04-30T06:30:05.440ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
64882025-04-30T06:30:05.442ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "552b5088-4776-4630-a169-b9d1363354da", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64892025-04-30T06:30:05.442ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
64902025-04-30T06:30:05.444ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10b519c0-6810-461c-9a59-32454b8ed4d4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64912025-04-30T06:30:05.444ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
64922025-04-30T06:30:05.457ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c202bc2-eca5-4f83-9cf3-6292aa17c13f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64932025-04-30T06:30:05.457ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
64942025-04-30T06:30:05.457ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ff86170-a5f2-4703-8884-92eaa6613ac9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64952025-04-30T06:30:05.457ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
64962025-04-30T06:30:05.457ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9a4e5d8-fc9e-4a78-bb78-5f59062f3807", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64972025-04-30T06:30:05.457ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
64982025-04-30T06:30:05.457ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3afc64bc-c5c2-4b48-9bdd-4895683d7f2f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
64992025-04-30T06:30:05.457ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
65002025-04-30T06:30:05.457ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1706015d-1109-40a2-ab67-c1ca80a852a8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65012025-04-30T06:30:05.457ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
65022025-04-30T06:30:05.459ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8af480c4-dee5-46a4-99f1-e5cb0eae1bfc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65032025-04-30T06:30:05.459ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
65042025-04-30T06:30:05.461ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bceb10fe-8196-40bd-81d5-1d0e0c2d74e5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65052025-04-30T06:30:05.461ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
65062025-04-30T06:30:05.462ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee05d537-1edb-47e0-b4f3-b52c5fb7e4ee", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65072025-04-30T06:30:05.462ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
65082025-04-30T06:30:05.463ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61f49336-2b29-4690-a1b6-288e56a2ed7c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65092025-04-30T06:30:05.463ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
65102025-04-30T06:30:05.464ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebd3dcd8-9105-4a36-ab7d-5c014a1fca4f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65112025-04-30T06:30:05.464ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
65122025-04-30T06:30:05.465ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb887d56-ac71-4f4e-a67b-0a74837048c7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65132025-04-30T06:30:05.465ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
65142025-04-30T06:30:05.467ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65152025-04-30T06:30:05.467ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65162025-04-30T06:30:05.467ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65172025-04-30T06:30:05.467ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
65182025-04-30T06:30:05.469ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65192025-04-30T06:30:05.469ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82e5d1fd-4598-4c27-8efa-82f5bf2feab2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65202025-04-30T06:30:05.469ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
65212025-04-30T06:30:05.472ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65222025-04-30T06:30:05.476ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65232025-04-30T06:30:05.476ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65242025-04-30T06:30:05.476ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
65252025-04-30T06:30:05.477ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65262025-04-30T06:30:05.480ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf1839dd-2668-436b-89a7-3963a3afb157", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65272025-04-30T06:30:05.480ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
65282025-04-30T06:30:05.480ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65292025-04-30T06:30:05.480ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
65302025-04-30T06:30:05.481ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65312025-04-30T06:30:05.482ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4b7b14e-b99f-4bd2-b8d8-97a1ddbab244", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65322025-04-30T06:30:05.482ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
65332025-04-30T06:30:05.485ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c5e4d20-b06e-4bda-adb1-3929ed201f00", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65342025-04-30T06:30:05.489ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
65352025-04-30T06:30:05.489ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a51907d-cea7-4464-a284-e02d216d0fba", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65362025-04-30T06:30:05.489ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
65372025-04-30T06:30:05.490ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cd70db4-1820-467a-a899-c5cbd1395baa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65382025-04-30T06:30:05.491ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
65392025-04-30T06:30:05.493ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd9396bd-99b8-483d-b806-1cb6ee993932", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65402025-04-30T06:30:05.493ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
65412025-04-30T06:30:05.494ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31e59b32-c5e9-4043-8642-cc3a2365f877", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65422025-04-30T06:30:05.494ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
65432025-04-30T06:30:05.497ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dad026a6-c24d-4ec5-bb45-c2b9126bc426", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65442025-04-30T06:30:05.498ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
65452025-04-30T06:30:05.498ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab991de0-dc92-433b-87c2-e68b1962a255", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65462025-04-30T06:30:05.499ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
65472025-04-30T06:30:05.501ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3be89e41-ebae-4f3c-bdae-f4fc8cc312b8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65482025-04-30T06:30:05.501ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
65492025-04-30T06:30:05.501ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65502025-04-30T06:30:05.501ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65512025-04-30T06:30:05.501ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65522025-04-30T06:30:05.502ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65532025-04-30T06:30:05.504ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65542025-04-30T06:30:05.509ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
65552025-04-30T06:30:05.510ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
65562025-04-30T06:30:05.510ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65572025-04-30T06:30:05.510ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65582025-04-30T06:30:05.510ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65592025-04-30T06:30:05.510ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65602025-04-30T06:30:05.513ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65612025-04-30T06:30:05.518ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
65622025-04-30T06:30:05.518ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65632025-04-30T06:30:05.518ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65642025-04-30T06:30:05.519ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65652025-04-30T06:30:05.611ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
65662025-04-30T06:30:05.611ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
65672025-04-30T06:30:05.614ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:38655 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
65682025-04-30T06:30:05.614ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65692025-04-30T06:30:05.614ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65702025-04-30T06:30:05.615ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
65712025-04-30T06:30:05.616ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:38655 req_id = 5904cefa-a9c1-4d20-ac08-d5ee7486e443 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
65722025-04-30T06:30:05.618ZINFOtest_image_promotion (dropshot): request completed latency_us = 2311 local_addr = [::1]:60070 method = GET remote_addr = [::1]:38655 req_id = 5904cefa-a9c1-4d20-ac08-d5ee7486e443 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
65732025-04-30T06:30:05.620ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65742025-04-30T06:30:05.620ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65752025-04-30T06:30:05.621ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65762025-04-30T06:30:05.621ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65772025-04-30T06:30:05.621ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5904cefa-a9c1-4d20-ac08-d5ee7486e443", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:04 GMT"} })
65782025-04-30T06:30:05.624ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
65792025-04-30T06:30:05.624ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
65802025-04-30T06:30:05.624ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
65812025-04-30T06:30:05.626ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
65822025-04-30T06:30:05.626ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65832025-04-30T06:30:05.626ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65842025-04-30T06:30:05.626ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65852025-04-30T06:30:05.627ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65862025-04-30T06:30:05.630ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65872025-04-30T06:30:05.643ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
65882025-04-30T06:30:05.645ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65892025-04-30T06:30:05.646ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65902025-04-30T06:30:05.647ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65912025-04-30T06:30:05.668ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65922025-04-30T06:30:05.668ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65932025-04-30T06:30:05.668ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65942025-04-30T06:30:05.668ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
65952025-04-30T06:30:05.670ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65962025-04-30T06:30:05.672ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65972025-04-30T06:30:05.705ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65982025-04-30T06:30:05.705ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65992025-04-30T06:30:05.705ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
66002025-04-30T06:30:05.706ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66012025-04-30T06:30:05.750ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
66022025-04-30T06:30:05.750ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
66032025-04-30T06:30:05.750ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
66042025-04-30T06:30:05.751ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
66052025-04-30T06:30:05.757ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
66062025-04-30T06:30:05.757ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
66072025-04-30T06:30:05.759ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
66082025-04-30T06:30:05.759ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
66092025-04-30T06:30:05.760ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 505.480648ms iteration = 19 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
66102025-04-30T06:30:05.760ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 20 reason = Signaled
66112025-04-30T06:30:05.760ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66122025-04-30T06:30:05.760ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66132025-04-30T06:30:05.760ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66142025-04-30T06:30:05.760ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66152025-04-30T06:30:05.760ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66162025-04-30T06:30:05.761ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66172025-04-30T06:30:05.761ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
66182025-04-30T06:30:05.761ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66192025-04-30T06:30:05.762ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
66202025-04-30T06:30:05.762ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
66212025-04-30T06:30:05.763ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66222025-04-30T06:30:05.767ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66232025-04-30T06:30:05.767ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66242025-04-30T06:30:05.767ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66252025-04-30T06:30:05.768ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66262025-04-30T06:30:05.772ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
66272025-04-30T06:30:05.772ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
66282025-04-30T06:30:05.773ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: af 64 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .d..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55388 req_id = 718a857e-f7df-4777-9d5f-194ff78e40f8
66292025-04-30T06:30:05.773ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44900,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55388 req_id = 718a857e-f7df-4777-9d5f-194ff78e40f8
66302025-04-30T06:30:05.774ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
66312025-04-30T06:30:05.774ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66322025-04-30T06:30:05.774ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55388 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 718a857e-f7df-4777-9d5f-194ff78e40f8
66332025-04-30T06:30:05.775ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3470, tv_nsec: 574424586 } })
66342025-04-30T06:30:05.775ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
66352025-04-30T06:30:05.776ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6c cb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e l...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52285 req_id = e12ca523-9361-4381-b6cb-8a3616fa2b39
66362025-04-30T06:30:05.776ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27851,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52285 req_id = e12ca523-9361-4381-b6cb-8a3616fa2b39
66372025-04-30T06:30:05.776ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66382025-04-30T06:30:05.776ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52285 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e12ca523-9361-4381-b6cb-8a3616fa2b39
66392025-04-30T06:30:05.860ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
66402025-04-30T06:30:05.861ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
66412025-04-30T06:30:05.862ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3f c6 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ?............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52119 req_id = e543a94b-94e2-46a8-a922-d462843fae28
66422025-04-30T06:30:05.862ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16326,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52119 req_id = e543a94b-94e2-46a8-a922-d462843fae28
66432025-04-30T06:30:05.862ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
66442025-04-30T06:30:05.863ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16326, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52119 req_id = e543a94b-94e2-46a8-a922-d462843fae28
66452025-04-30T06:30:05.864ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
66462025-04-30T06:30:05.864ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
66472025-04-30T06:30:05.864ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
66482025-04-30T06:30:05.870ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ba3d780-c1eb-4d4b-9829-fd27db09ad50", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66492025-04-30T06:30:05.870ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
66502025-04-30T06:30:05.872ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2593d356-67b4-43fe-bd53-41d4c7939b08", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66512025-04-30T06:30:05.872ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
66522025-04-30T06:30:05.878ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1d5a18c-756f-4f53-b4ee-032705243a7c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66532025-04-30T06:30:05.878ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
66542025-04-30T06:30:05.884ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "edf45267-a6cc-423c-8e63-dab19a461121", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66552025-04-30T06:30:05.884ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
66562025-04-30T06:30:05.884ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
66572025-04-30T06:30:05.890ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c67c47d5-15cf-448c-a6a3-893f2cf19079", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66582025-04-30T06:30:05.890ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
66592025-04-30T06:30:05.894ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aafe60f5-78e3-40f3-84a9-b4bfd3ec0daa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66602025-04-30T06:30:05.894ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
66612025-04-30T06:30:05.904ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4185a4df-5c4f-4442-b623-ab4c2655010e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66622025-04-30T06:30:05.904ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
66632025-04-30T06:30:05.906ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24af8283-b22d-4bfe-a948-66b30a6b507f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66642025-04-30T06:30:05.906ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
66652025-04-30T06:30:05.907ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6fa113f-423a-4094-abc5-41296196da38", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66662025-04-30T06:30:05.908ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
66672025-04-30T06:30:05.922ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c496e19a-e029-48fd-aabb-6064c4ebc6ee", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66682025-04-30T06:30:05.922ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
66692025-04-30T06:30:05.923ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb52973d-5890-488c-953c-b3b82fdb6ae5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66702025-04-30T06:30:05.923ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
66712025-04-30T06:30:05.925ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b224b201-474e-45c7-8a4b-3d484828177f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66722025-04-30T06:30:05.926ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
66732025-04-30T06:30:05.929ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18b6f138-8832-48b1-9045-9f0f4c2fd87b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66742025-04-30T06:30:05.929ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
66752025-04-30T06:30:05.935ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56831820-f8be-47b6-a9a6-2ff336fc4b41", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66762025-04-30T06:30:05.935ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
66772025-04-30T06:30:05.935ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66782025-04-30T06:30:05.936ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66792025-04-30T06:30:05.936ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66802025-04-30T06:30:05.936ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
66812025-04-30T06:30:05.937ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66822025-04-30T06:30:05.940ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66832025-04-30T06:30:05.951ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66842025-04-30T06:30:05.951ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66852025-04-30T06:30:05.951ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
66862025-04-30T06:30:05.952ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66872025-04-30T06:30:05.958ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66882025-04-30T06:30:05.958ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
66892025-04-30T06:30:05.958ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66902025-04-30T06:30:05.992ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c10a2dc1-c13f-4bce-9c29-9a296cad0d2f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66912025-04-30T06:30:05.992ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
66922025-04-30T06:30:06.023ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "381b6758-85fa-4dad-a96a-02fe80b8d078", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66932025-04-30T06:30:06.023ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
66942025-04-30T06:30:06.027ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d1f4f33-ac04-47a2-ac5d-fccd3def52e0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66952025-04-30T06:30:06.027ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
66962025-04-30T06:30:06.037ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a799b064-fd10-4617-b8a4-000e95988a6f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
66972025-04-30T06:30:06.038ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
66982025-04-30T06:30:06.038ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
66992025-04-30T06:30:06.043ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64956ee7-f48c-4580-bd73-aadda8451c2a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67002025-04-30T06:30:06.043ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
67012025-04-30T06:30:06.044ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "853b451b-dbfc-47ac-afb4-f4cf7bcc2d89", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67022025-04-30T06:30:06.044ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
67032025-04-30T06:30:06.046ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bad5ec26-11d2-458a-b2a4-237e6c93af34", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67042025-04-30T06:30:06.046ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
67052025-04-30T06:30:06.047ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9f9a0fe-ef3a-4afb-b826-634908b91860", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67062025-04-30T06:30:06.048ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
67072025-04-30T06:30:06.050ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85477358-6a36-4940-bc8d-05e456632b5b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67082025-04-30T06:30:06.050ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
67092025-04-30T06:30:06.051ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f371d2a5-c90e-4f47-bd40-62d5306e2c16", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67102025-04-30T06:30:06.052ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
67112025-04-30T06:30:06.054ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9e7d456-13d7-4e0a-8a2a-30ac8894489d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67122025-04-30T06:30:06.054ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
67132025-04-30T06:30:06.056ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7440029e-b3ad-49a1-a231-bce5e2be8626", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67142025-04-30T06:30:06.056ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
67152025-04-30T06:30:06.058ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc6cada9-a051-4336-84b8-1ba34ee64849", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67162025-04-30T06:30:06.059ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
67172025-04-30T06:30:06.062ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f83584f9-2bf9-4e87-be5e-349bbedc6a85", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67182025-04-30T06:30:06.062ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
67192025-04-30T06:30:06.064ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ca9aca6-f77c-4370-ba4a-47b8b84cc742", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67202025-04-30T06:30:06.064ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
67212025-04-30T06:30:06.066ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8429226a-43ce-426e-88f2-af3ea2b58952", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67222025-04-30T06:30:06.067ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
67232025-04-30T06:30:06.068ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2eca5f69-3ed0-48ba-b9d1-e466ab3014d4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67242025-04-30T06:30:06.068ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
67252025-04-30T06:30:06.069ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "11cf5418-49dc-45b0-8438-97acccbe1a24", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67262025-04-30T06:30:06.069ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
67272025-04-30T06:30:06.070ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0bcc3a4-1fb3-4d4d-ac46-d6f9cabb6e02", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67282025-04-30T06:30:06.071ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
67292025-04-30T06:30:06.072ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c83021-dba0-4a52-94ec-82ca598d00c2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67302025-04-30T06:30:06.072ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
67312025-04-30T06:30:06.073ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "420996aa-ba77-493e-b45f-c4c444993abd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67322025-04-30T06:30:06.074ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
67332025-04-30T06:30:06.078ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a36068f5-1cb6-4ccd-9f95-9ee99a8c6677", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67342025-04-30T06:30:06.078ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
67352025-04-30T06:30:06.082ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48b82d54-acda-4c0c-9f9b-0bc8882fd5eb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67362025-04-30T06:30:06.082ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
67372025-04-30T06:30:06.086ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbcd9f7c-2fdf-404e-a7c9-63222fa4b492", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67382025-04-30T06:30:06.087ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
67392025-04-30T06:30:06.090ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00d38504-9d7a-456e-a8cd-9fc79032c971", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67402025-04-30T06:30:06.090ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
67412025-04-30T06:30:06.090ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
67422025-04-30T06:30:06.090ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67432025-04-30T06:30:06.090ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67442025-04-30T06:30:06.090ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
67452025-04-30T06:30:06.091ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67462025-04-30T06:30:06.094ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7171d3e-5e16-4206-affb-12105d2d2201", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67472025-04-30T06:30:06.095ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
67482025-04-30T06:30:06.095ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67492025-04-30T06:30:06.109ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
67502025-04-30T06:30:06.109ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67512025-04-30T06:30:06.109ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
67522025-04-30T06:30:06.112ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67532025-04-30T06:30:06.118ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67542025-04-30T06:30:06.118ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
67552025-04-30T06:30:06.120ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67562025-04-30T06:30:06.124ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd53ea95-f0bd-4ae3-855f-d6868cba8a7c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67572025-04-30T06:30:06.124ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
67582025-04-30T06:30:06.128ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48188836-880a-411f-8a90-c6049784ccf0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67592025-04-30T06:30:06.130ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
67602025-04-30T06:30:06.132ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc6006ee-ffc2-4428-91cf-f231f2178176", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67612025-04-30T06:30:06.132ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
67622025-04-30T06:30:06.133ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e12abdcf-9192-4348-91ac-0080db1ddb12", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67632025-04-30T06:30:06.134ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
67642025-04-30T06:30:06.135ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c97a8fb-99b8-4dd1-a120-c30e4359cf37", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67652025-04-30T06:30:06.136ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
67662025-04-30T06:30:06.136ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67672025-04-30T06:30:06.136ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67682025-04-30T06:30:06.136ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67692025-04-30T06:30:06.137ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67702025-04-30T06:30:06.153ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67712025-04-30T06:30:06.166ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
67722025-04-30T06:30:06.166ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
67732025-04-30T06:30:06.166ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67742025-04-30T06:30:06.166ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67752025-04-30T06:30:06.166ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67762025-04-30T06:30:06.167ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67772025-04-30T06:30:06.173ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67782025-04-30T06:30:06.186ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
67792025-04-30T06:30:06.186ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67802025-04-30T06:30:06.186ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67812025-04-30T06:30:06.186ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67822025-04-30T06:30:06.242ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
67832025-04-30T06:30:06.242ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
67842025-04-30T06:30:06.243ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:48141 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
67852025-04-30T06:30:06.244ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:48141 req_id = 36a2b417-bea0-492b-b859-a8839d2c8825 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
67862025-04-30T06:30:06.244ZINFOtest_image_promotion (dropshot): request completed latency_us = 803 local_addr = [::1]:60070 method = GET remote_addr = [::1]:48141 req_id = 36a2b417-bea0-492b-b859-a8839d2c8825 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
67872025-04-30T06:30:06.246ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "36a2b417-bea0-492b-b859-a8839d2c8825", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
67882025-04-30T06:30:06.251ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
67892025-04-30T06:30:06.251ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
67902025-04-30T06:30:06.251ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
67912025-04-30T06:30:06.255ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
67922025-04-30T06:30:06.256ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67932025-04-30T06:30:06.256ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67942025-04-30T06:30:06.256ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67952025-04-30T06:30:06.256ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67962025-04-30T06:30:06.263ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67972025-04-30T06:30:06.269ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
67982025-04-30T06:30:06.270ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67992025-04-30T06:30:06.270ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68002025-04-30T06:30:06.270ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68012025-04-30T06:30:06.271ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
68022025-04-30T06:30:06.301ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
68032025-04-30T06:30:06.301ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
68042025-04-30T06:30:06.301ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 541.316608ms iteration = 20 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
68052025-04-30T06:30:06.301ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 21 reason = Signaled
68062025-04-30T06:30:06.301ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68072025-04-30T06:30:06.302ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68082025-04-30T06:30:06.302ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68092025-04-30T06:30:06.302ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68102025-04-30T06:30:06.303ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68112025-04-30T06:30:06.306ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68122025-04-30T06:30:06.312ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68132025-04-30T06:30:06.312ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68142025-04-30T06:30:06.312ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68152025-04-30T06:30:06.312ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68162025-04-30T06:30:06.318ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
68172025-04-30T06:30:06.318ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
68182025-04-30T06:30:06.320ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: eb cd 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49913 req_id = ad0bec20-583c-4164-aed3-fca7c94633c4
68192025-04-30T06:30:06.320ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60365,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49913 req_id = ad0bec20-583c-4164-aed3-fca7c94633c4
68202025-04-30T06:30:06.320ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
68212025-04-30T06:30:06.320ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68222025-04-30T06:30:06.322ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49913 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ad0bec20-583c-4164-aed3-fca7c94633c4
68232025-04-30T06:30:06.322ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3471, tv_nsec: 121531410 } })
68242025-04-30T06:30:06.322ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
68252025-04-30T06:30:06.323ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68262025-04-30T06:30:06.323ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68272025-04-30T06:30:06.323ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68282025-04-30T06:30:06.323ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
68292025-04-30T06:30:06.327ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68302025-04-30T06:30:06.327ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5c ca 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e \\...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65100 req_id = 711788aa-b5ee-484d-aeb7-bc27b8332fd5
68312025-04-30T06:30:06.327ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23754,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65100 req_id = 711788aa-b5ee-484d-aeb7-bc27b8332fd5
68322025-04-30T06:30:06.327ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68332025-04-30T06:30:06.328ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65100 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 711788aa-b5ee-484d-aeb7-bc27b8332fd5
68342025-04-30T06:30:06.460ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
68352025-04-30T06:30:06.460ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68362025-04-30T06:30:06.466ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68372025-04-30T06:30:06.466ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68382025-04-30T06:30:06.466ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
68392025-04-30T06:30:06.467ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68402025-04-30T06:30:06.471ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 37 e4 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 7............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50761 req_id = 2b06971d-9ddd-4a3f-87da-e2e685a1888c
68412025-04-30T06:30:06.472ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14308,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50761 req_id = 2b06971d-9ddd-4a3f-87da-e2e685a1888c
68422025-04-30T06:30:06.472ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
68432025-04-30T06:30:06.474ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14308, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50761 req_id = 2b06971d-9ddd-4a3f-87da-e2e685a1888c
68442025-04-30T06:30:06.474ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68452025-04-30T06:30:06.474ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
68462025-04-30T06:30:06.477ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68472025-04-30T06:30:06.477ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
68482025-04-30T06:30:06.477ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
68492025-04-30T06:30:06.478ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
68502025-04-30T06:30:06.483ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "65c5d95b-7c3a-4499-9ab7-d4ec289ac110", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68512025-04-30T06:30:06.484ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
68522025-04-30T06:30:06.485ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3816ad3-b0d6-4025-98e0-91fb527aeca2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68532025-04-30T06:30:06.485ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
68542025-04-30T06:30:06.489ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "baa1f0ef-0edd-44f3-af5d-55a179515327", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68552025-04-30T06:30:06.490ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
68562025-04-30T06:30:06.493ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d897d447-4c15-4a06-9dd1-a3f15db8e37a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68572025-04-30T06:30:06.493ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
68582025-04-30T06:30:06.496ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cc5a0a9-dcd7-4c42-9b68-388c2d782792", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68592025-04-30T06:30:06.496ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
68602025-04-30T06:30:06.500ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e7e7467-24c1-4a06-89f5-02ff02a0422b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68612025-04-30T06:30:06.500ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
68622025-04-30T06:30:06.503ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e020f716-09b1-4028-bcab-b20bb2310459", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68632025-04-30T06:30:06.503ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
68642025-04-30T06:30:06.504ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
68652025-04-30T06:30:06.505ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
68662025-04-30T06:30:06.506ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 92270553-3601-4feb-82a5-e01c8984ea92 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
68672025-04-30T06:30:06.507ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 92270553-3601-4feb-82a5-e01c8984ea92 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
68682025-04-30T06:30:06.507ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 92270553-3601-4feb-82a5-e01c8984ea92 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
68692025-04-30T06:30:06.508ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 92270553-3601-4feb-82a5-e01c8984ea92 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
68702025-04-30T06:30:06.511ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
68712025-04-30T06:30:06.511ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "015a8f47-1235-4dbf-bbb6-8c7140e75987", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68722025-04-30T06:30:06.511ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
68732025-04-30T06:30:06.512ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 6001 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 92270553-3601-4feb-82a5-e01c8984ea92 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
68742025-04-30T06:30:06.513ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "92270553-3601-4feb-82a5-e01c8984ea92", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68752025-04-30T06:30:06.513ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
68762025-04-30T06:30:06.514ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 264809d6-9b7d-49b5-8d8e-43b81ccde552 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
68772025-04-30T06:30:06.514ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 264809d6-9b7d-49b5-8d8e-43b81ccde552 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
68782025-04-30T06:30:06.514ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 264809d6-9b7d-49b5-8d8e-43b81ccde552 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
68792025-04-30T06:30:06.515ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 264809d6-9b7d-49b5-8d8e-43b81ccde552 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
68802025-04-30T06:30:06.515ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39e02e89-c359-4dbb-9f0d-5f5dd606cc6e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68812025-04-30T06:30:06.516ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
68822025-04-30T06:30:06.517ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a5704c6-9888-42b8-8fe9-b5eaf629c1c0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68832025-04-30T06:30:06.517ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
68842025-04-30T06:30:06.518ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 3965 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 264809d6-9b7d-49b5-8d8e-43b81ccde552 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
68852025-04-30T06:30:06.518ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "264809d6-9b7d-49b5-8d8e-43b81ccde552", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68862025-04-30T06:30:06.518ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68872025-04-30T06:30:06.519ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
68882025-04-30T06:30:06.519ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
68892025-04-30T06:30:06.520ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07492227-b85a-489e-bc59-cd2f68413ce7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68902025-04-30T06:30:06.520ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
68912025-04-30T06:30:06.522ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdbf6c10-43c4-4c83-8d4b-dd8627d194a3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68922025-04-30T06:30:06.522ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
68932025-04-30T06:30:06.523ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a30e71f3-224c-4088-b0a2-a4de6f47a28d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68942025-04-30T06:30:06.524ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
68952025-04-30T06:30:06.526ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be1c145c-46c1-4cae-a58d-e5175c3da84e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68962025-04-30T06:30:06.529ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
68972025-04-30T06:30:06.533ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad11a2e8-0f26-4a69-9996-d7e050740bd7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
68982025-04-30T06:30:06.533ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
68992025-04-30T06:30:06.535ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8a49cbf-4839-412f-8854-a7c44503cf7d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69002025-04-30T06:30:06.535ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
69012025-04-30T06:30:06.537ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ea65c71-6cf9-4afe-bf7f-205a93ea2809", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69022025-04-30T06:30:06.538ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
69032025-04-30T06:30:06.538ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
69042025-04-30T06:30:06.538ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
69052025-04-30T06:30:06.540ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
69062025-04-30T06:30:06.541ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
69072025-04-30T06:30:06.541ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
69082025-04-30T06:30:06.541ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
69092025-04-30T06:30:06.542ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afeddf26-6cc3-4633-b76e-b95c900242ca", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69102025-04-30T06:30:06.542ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
69112025-04-30T06:30:06.545ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e38405bc-9510-4303-b956-1447166902c8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69122025-04-30T06:30:06.545ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
69132025-04-30T06:30:06.547ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffd5fa0e-1b58-4afd-92b2-41d0e6559cc8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69142025-04-30T06:30:06.547ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
69152025-04-30T06:30:06.549ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20568658-b163-42e0-b16a-8196a604e1bb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69162025-04-30T06:30:06.550ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
69172025-04-30T06:30:06.550ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9881707b-196d-484b-ad97-a3d6bd36e15f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69182025-04-30T06:30:06.551ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
69192025-04-30T06:30:06.552ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "47c44be4-d3b1-429a-b5a5-4ac7c04f8c1f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69202025-04-30T06:30:06.552ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
69212025-04-30T06:30:06.554ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7ff7609-3b4c-4e42-8b7e-c8f870589183", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69222025-04-30T06:30:06.554ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
69232025-04-30T06:30:06.560ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38f85e58-f95e-4ff3-a237-1e461f0615da", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69242025-04-30T06:30:06.560ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
69252025-04-30T06:30:06.562ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69262025-04-30T06:30:06.562ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69272025-04-30T06:30:06.562ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69282025-04-30T06:30:06.562ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
69292025-04-30T06:30:06.562ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69302025-04-30T06:30:06.565ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69312025-04-30T06:30:06.573ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69322025-04-30T06:30:06.573ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69332025-04-30T06:30:06.573ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
69342025-04-30T06:30:06.573ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69352025-04-30T06:30:06.575ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d780e181-6cf5-4c1b-bab5-58e996666a2c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69362025-04-30T06:30:06.575ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
69372025-04-30T06:30:06.576ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69382025-04-30T06:30:06.576ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
69392025-04-30T06:30:06.577ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69402025-04-30T06:30:06.581ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "041d3ba9-adc6-4bdb-aa57-1d03596dfcce", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69412025-04-30T06:30:06.581ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
69422025-04-30T06:30:06.583ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd2b3612-7fbd-47ad-91b3-91cca2c4a8bc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69432025-04-30T06:30:06.583ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
69442025-04-30T06:30:06.586ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44139bfe-bbed-488f-a0e2-6b5ae14e98f8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69452025-04-30T06:30:06.587ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
69462025-04-30T06:30:06.592ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78520b68-3371-4cf5-bbfa-8dd2bf7470aa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69472025-04-30T06:30:06.592ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
69482025-04-30T06:30:06.594ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d1820c1-b733-4038-9a3a-6c6950e861c4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69492025-04-30T06:30:06.594ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
69502025-04-30T06:30:06.598ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e58fe681-779e-453d-a0e6-4b0582764336", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69512025-04-30T06:30:06.598ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
69522025-04-30T06:30:06.599ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62909b6a-e5ad-4f59-af63-71b5d2253bb2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69532025-04-30T06:30:06.599ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
69542025-04-30T06:30:06.601ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eda7785c-7b5a-41a6-89e8-62a2ab083a88", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69552025-04-30T06:30:06.602ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
69562025-04-30T06:30:06.609ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3465821c-132b-42da-b582-793442a788a9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69572025-04-30T06:30:06.609ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
69582025-04-30T06:30:06.610ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
69592025-04-30T06:30:06.615ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ce4d18e-ac84-481d-9c39-d40d3a85a7f8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69602025-04-30T06:30:06.615ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
69612025-04-30T06:30:06.619ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1ab2b9e-feaf-4ff1-8bcd-cf8fa2a575d6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69622025-04-30T06:30:06.619ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
69632025-04-30T06:30:06.623ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56a113fa-967f-427e-9651-366a0c7a6f24", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69642025-04-30T06:30:06.623ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
69652025-04-30T06:30:06.625ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5493848-f517-4121-bbec-f8cdd350a1ab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69662025-04-30T06:30:06.625ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
69672025-04-30T06:30:06.629ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbd9ee30-17bb-474c-af1b-edff8e4ae538", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69682025-04-30T06:30:06.629ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
69692025-04-30T06:30:06.631ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57705ff2-1b05-4e5f-ac85-7ebc46db7cc9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69702025-04-30T06:30:06.631ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
69712025-04-30T06:30:06.632ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7755e07-b64e-47d3-ab75-d171b073b5bc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69722025-04-30T06:30:06.633ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
69732025-04-30T06:30:06.634ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7c907ad-45aa-412e-8331-65fbb15007f2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69742025-04-30T06:30:06.635ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
69752025-04-30T06:30:06.637ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e461b1ab-c7c3-401c-8e93-0f63bf9f455a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69762025-04-30T06:30:06.637ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
69772025-04-30T06:30:06.639ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "587a6d50-54f5-43e6-822a-2c3a9056f7e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
69782025-04-30T06:30:06.639ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
69792025-04-30T06:30:06.639ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69802025-04-30T06:30:06.639ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69812025-04-30T06:30:06.639ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69822025-04-30T06:30:06.640ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69832025-04-30T06:30:06.645ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69842025-04-30T06:30:06.663ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
69852025-04-30T06:30:06.663ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
69862025-04-30T06:30:06.663ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69872025-04-30T06:30:06.663ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69882025-04-30T06:30:06.663ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69892025-04-30T06:30:06.664ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69902025-04-30T06:30:06.665ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69912025-04-30T06:30:06.665ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69922025-04-30T06:30:06.665ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69932025-04-30T06:30:06.665ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
69942025-04-30T06:30:06.665ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69952025-04-30T06:30:06.668ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69962025-04-30T06:30:06.675ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
69972025-04-30T06:30:06.675ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69982025-04-30T06:30:06.675ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69992025-04-30T06:30:06.675ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70002025-04-30T06:30:06.678ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70012025-04-30T06:30:06.687ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70022025-04-30T06:30:06.688ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70032025-04-30T06:30:06.688ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
70042025-04-30T06:30:06.688ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70052025-04-30T06:30:06.693ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70062025-04-30T06:30:06.694ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
70072025-04-30T06:30:06.695ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70082025-04-30T06:30:06.775ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
70092025-04-30T06:30:06.775ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
70102025-04-30T06:30:06.777ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:39196 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
70112025-04-30T06:30:06.777ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
70122025-04-30T06:30:06.777ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
70132025-04-30T06:30:06.778ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
70142025-04-30T06:30:06.779ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
70152025-04-30T06:30:06.779ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:39196 req_id = e1c3356b-59d2-42b1-8eee-324ef22b7eba sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70162025-04-30T06:30:06.780ZINFOtest_image_promotion (dropshot): request completed latency_us = 448 local_addr = [::1]:60070 method = GET remote_addr = [::1]:39196 req_id = e1c3356b-59d2-42b1-8eee-324ef22b7eba response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70172025-04-30T06:30:06.780ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1c3356b-59d2-42b1-8eee-324ef22b7eba", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:05 GMT"} })
70182025-04-30T06:30:06.783ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70192025-04-30T06:30:06.783ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70202025-04-30T06:30:06.783ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
70212025-04-30T06:30:06.786ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70222025-04-30T06:30:06.786ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70232025-04-30T06:30:06.786ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70242025-04-30T06:30:06.786ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70252025-04-30T06:30:06.787ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70262025-04-30T06:30:06.790ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
70272025-04-30T06:30:06.790ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
70282025-04-30T06:30:06.790ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70292025-04-30T06:30:06.794ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70302025-04-30T06:30:06.794ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70312025-04-30T06:30:06.794ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70322025-04-30T06:30:06.795ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70332025-04-30T06:30:06.802ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
70342025-04-30T06:30:06.802ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
70352025-04-30T06:30:06.822ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
70362025-04-30T06:30:06.833ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
70372025-04-30T06:30:06.833ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
70382025-04-30T06:30:06.833ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 532.044363ms iteration = 21 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
70392025-04-30T06:30:06.834ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 22 reason = Signaled
70402025-04-30T06:30:06.834ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70412025-04-30T06:30:06.834ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70422025-04-30T06:30:06.834ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70432025-04-30T06:30:06.834ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70442025-04-30T06:30:06.836ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70452025-04-30T06:30:06.840ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70462025-04-30T06:30:06.844ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70472025-04-30T06:30:06.844ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70482025-04-30T06:30:06.845ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70492025-04-30T06:30:06.846ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70502025-04-30T06:30:06.849ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
70512025-04-30T06:30:06.849ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
70522025-04-30T06:30:06.850ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0c fa 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57917 req_id = 544aef9a-1501-43cf-b635-499ea18f7e22
70532025-04-30T06:30:06.850ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3322,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57917 req_id = 544aef9a-1501-43cf-b635-499ea18f7e22
70542025-04-30T06:30:06.850ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
70552025-04-30T06:30:06.850ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
70562025-04-30T06:30:06.850ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57917 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 544aef9a-1501-43cf-b635-499ea18f7e22
70572025-04-30T06:30:06.851ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3471, tv_nsec: 649930731 } })
70582025-04-30T06:30:06.851ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
70592025-04-30T06:30:06.852ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 84 b3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59999 req_id = c68f59df-d9e4-4f91-8e11-abc6242fe7a7
70602025-04-30T06:30:06.852ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33971,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59999 req_id = c68f59df-d9e4-4f91-8e11-abc6242fe7a7
70612025-04-30T06:30:06.853ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
70622025-04-30T06:30:06.853ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59999 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c68f59df-d9e4-4f91-8e11-abc6242fe7a7
70632025-04-30T06:30:06.933ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
70642025-04-30T06:30:06.933ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70652025-04-30T06:30:06.933ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70662025-04-30T06:30:06.933ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70672025-04-30T06:30:06.934ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
70682025-04-30T06:30:06.934ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70692025-04-30T06:30:06.935ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
70702025-04-30T06:30:06.937ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 61 b5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c a............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62650 req_id = 5c718e0d-e8d8-41c0-b365-e01672a9b9ed
70712025-04-30T06:30:06.937ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25013,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62650 req_id = 5c718e0d-e8d8-41c0-b365-e01672a9b9ed
70722025-04-30T06:30:06.937ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
70732025-04-30T06:30:06.937ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25013, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62650 req_id = 5c718e0d-e8d8-41c0-b365-e01672a9b9ed
70742025-04-30T06:30:06.937ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
70752025-04-30T06:30:06.937ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
70762025-04-30T06:30:06.937ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
70772025-04-30T06:30:06.939ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "620800f6-eaff-4c2c-af13-21528dde2524", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
70782025-04-30T06:30:06.939ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
70792025-04-30T06:30:06.939ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70802025-04-30T06:30:06.946ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70812025-04-30T06:30:06.946ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70822025-04-30T06:30:06.946ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
70832025-04-30T06:30:06.947ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70842025-04-30T06:30:06.951ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5f45d1e-559d-4c28-8d11-0f33d25aee2f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
70852025-04-30T06:30:06.951ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
70862025-04-30T06:30:06.952ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70872025-04-30T06:30:06.952ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
70882025-04-30T06:30:06.952ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70892025-04-30T06:30:06.991ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c24e59ac-2a2c-45d1-ba4e-f0c146ff2e39", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
70902025-04-30T06:30:06.991ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
70912025-04-30T06:30:06.994ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5028b562-68a4-45b0-a996-fd40d07569a9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
70922025-04-30T06:30:06.994ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
70932025-04-30T06:30:07.008ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2779be5-dfe4-42e6-bb40-a7c7379aab0b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
70942025-04-30T06:30:07.009ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
70952025-04-30T06:30:07.011ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85ac53bc-1826-4fd8-ace9-5a0fce86e252", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
70962025-04-30T06:30:07.011ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
70972025-04-30T06:30:07.015ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
70982025-04-30T06:30:07.017ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fd1f755-4589-4c37-ba9f-e66099d83605", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
70992025-04-30T06:30:07.017ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
71002025-04-30T06:30:07.020ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b22ac18a-a41a-4258-abb1-de9091c776b3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71012025-04-30T06:30:07.021ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
71022025-04-30T06:30:07.023ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83d7ee5f-66b3-4360-96a4-fdb614b4104a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71032025-04-30T06:30:07.024ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
71042025-04-30T06:30:07.026ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c231aa5-0add-4bf4-ad9f-99e1e699002f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71052025-04-30T06:30:07.028ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
71062025-04-30T06:30:07.029ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cb10157-9c33-4370-a1ea-ee7371d6042f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71072025-04-30T06:30:07.029ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
71082025-04-30T06:30:07.031ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3afe80fb-5c1f-44ef-b879-10128305ff4e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71092025-04-30T06:30:07.031ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
71102025-04-30T06:30:07.032ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35180080-ed15-44dc-b09d-8cd38b6980a4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71112025-04-30T06:30:07.033ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
71122025-04-30T06:30:07.036ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3de3c41d-f95c-4e74-8b2a-d673b15bb1fa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71132025-04-30T06:30:07.037ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
71142025-04-30T06:30:07.040ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ebdea6c-fee5-4061-b8c4-6320c12b0cc2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71152025-04-30T06:30:07.040ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
71162025-04-30T06:30:07.042ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "edac4b4d-130c-40bf-8cfc-ccad0957916b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71172025-04-30T06:30:07.042ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
71182025-04-30T06:30:07.044ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d39278d-d449-4e9d-8a30-ffb96a26d20a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71192025-04-30T06:30:07.045ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
71202025-04-30T06:30:07.048ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a8f288d-2b19-4f92-8646-bc1c854aef92", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71212025-04-30T06:30:07.048ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
71222025-04-30T06:30:07.050ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "003d1f45-5fab-46aa-bf8f-5237bfc7af8d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71232025-04-30T06:30:07.050ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
71242025-04-30T06:30:07.051ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5dd2fe1-5d9c-4233-90fe-3ce354ff70fb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71252025-04-30T06:30:07.052ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
71262025-04-30T06:30:07.053ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4659e9d0-3912-4b7e-9ce0-bd2fe754d4b6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71272025-04-30T06:30:07.053ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
71282025-04-30T06:30:07.055ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42ba9465-8e7b-4e35-a669-a13e2626e8d3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71292025-04-30T06:30:07.055ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
71302025-04-30T06:30:07.056ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d82c25e1-37e2-4975-9e89-7e027e37e24b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71312025-04-30T06:30:07.056ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
71322025-04-30T06:30:07.062ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e902b35-42e8-4240-9b8c-196acc8e0d0f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71332025-04-30T06:30:07.062ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
71342025-04-30T06:30:07.068ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71352025-04-30T06:30:07.069ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71362025-04-30T06:30:07.069ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71372025-04-30T06:30:07.069ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
71382025-04-30T06:30:07.071ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71392025-04-30T06:30:07.072ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6d528c2-4b9c-4405-a063-e8e3b1543144", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71402025-04-30T06:30:07.072ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
71412025-04-30T06:30:07.076ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71422025-04-30T06:30:07.087ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71432025-04-30T06:30:07.087ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71442025-04-30T06:30:07.087ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
71452025-04-30T06:30:07.089ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71462025-04-30T06:30:07.091ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0351ec6-c9fd-41df-95b3-696520446301", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71472025-04-30T06:30:07.092ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
71482025-04-30T06:30:07.092ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71492025-04-30T06:30:07.092ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
71502025-04-30T06:30:07.093ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71512025-04-30T06:30:07.097ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c5ae9c1-6e1d-499b-9968-eaa61e064574", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71522025-04-30T06:30:07.097ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
71532025-04-30T06:30:07.102ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28d24465-58de-49cc-90c4-b893e46c5f62", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71542025-04-30T06:30:07.102ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
71552025-04-30T06:30:07.108ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ff3ac43-ac9f-40a2-bcac-881bda462ed9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71562025-04-30T06:30:07.108ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
71572025-04-30T06:30:07.114ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1060c378-3677-4b9d-9593-bb8421d19c39", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71582025-04-30T06:30:07.115ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
71592025-04-30T06:30:07.116ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c83b049b-1d92-49d1-8fb8-75870e07a86b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71602025-04-30T06:30:07.116ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
71612025-04-30T06:30:07.122ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
71622025-04-30T06:30:07.122ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcab76e2-e23d-41cc-a239-16c52a5c82c1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71632025-04-30T06:30:07.123ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
71642025-04-30T06:30:07.125ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e846cfb8-1028-4ae7-bd81-9d5733e2d9d1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71652025-04-30T06:30:07.125ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
71662025-04-30T06:30:07.127ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c8a5ec3-d2d1-45cb-9c07-d0494ade3ec2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71672025-04-30T06:30:07.127ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
71682025-04-30T06:30:07.130ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91f479b9-f06b-47f1-82d8-b6e59d41014f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71692025-04-30T06:30:07.130ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
71702025-04-30T06:30:07.134ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1125013-7639-428f-bf87-16ccf74cbf53", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71712025-04-30T06:30:07.134ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
71722025-04-30T06:30:07.138ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40808de0-cf40-4fc3-a3e9-c31a7afd58fa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71732025-04-30T06:30:07.138ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
71742025-04-30T06:30:07.142ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e4ae2ce-f862-47c8-8680-73758c8d23d9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71752025-04-30T06:30:07.142ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
71762025-04-30T06:30:07.146ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f5b0ff6-4368-4980-8049-14a4721465ba", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71772025-04-30T06:30:07.146ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
71782025-04-30T06:30:07.151ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a19c0de4-be78-4622-9aa6-92c1fd44f6f7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71792025-04-30T06:30:07.151ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
71802025-04-30T06:30:07.161ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36ea6bfb-f8dd-4200-841e-7952eaa0a29c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71812025-04-30T06:30:07.161ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
71822025-04-30T06:30:07.162ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5a26147-d5c9-490a-9937-2c197ff5ce02", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71832025-04-30T06:30:07.163ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
71842025-04-30T06:30:07.164ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ddbbacc-60f0-435a-8256-46944e6c6933", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71852025-04-30T06:30:07.164ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
71862025-04-30T06:30:07.166ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd96602a-25ea-4485-bf5f-b3c131d4b64b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71872025-04-30T06:30:07.166ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
71882025-04-30T06:30:07.168ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87d0e1cc-b636-4baa-b671-66295ae2c0e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
71892025-04-30T06:30:07.169ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
71902025-04-30T06:30:07.169ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71912025-04-30T06:30:07.169ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71922025-04-30T06:30:07.169ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71932025-04-30T06:30:07.170ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71942025-04-30T06:30:07.179ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71952025-04-30T06:30:07.179ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71962025-04-30T06:30:07.179ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71972025-04-30T06:30:07.179ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
71982025-04-30T06:30:07.180ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71992025-04-30T06:30:07.181ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72002025-04-30T06:30:07.191ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72012025-04-30T06:30:07.191ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72022025-04-30T06:30:07.191ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72032025-04-30T06:30:07.191ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72042025-04-30T06:30:07.193ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72052025-04-30T06:30:07.194ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72062025-04-30T06:30:07.197ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72072025-04-30T06:30:07.209ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72082025-04-30T06:30:07.209ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72092025-04-30T06:30:07.209ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
72102025-04-30T06:30:07.210ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72112025-04-30T06:30:07.210ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72122025-04-30T06:30:07.213ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72132025-04-30T06:30:07.214ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72142025-04-30T06:30:07.214ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72152025-04-30T06:30:07.214ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72162025-04-30T06:30:07.216ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72172025-04-30T06:30:07.217ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
72182025-04-30T06:30:07.220ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72192025-04-30T06:30:07.292ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
72202025-04-30T06:30:07.292ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
72212025-04-30T06:30:07.292ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:53097 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
72222025-04-30T06:30:07.293ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:53097 req_id = 0147bc93-3b84-4b49-b191-d50ea87c9eb5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
72232025-04-30T06:30:07.293ZINFOtest_image_promotion (dropshot): request completed latency_us = 749 local_addr = [::1]:60070 method = GET remote_addr = [::1]:53097 req_id = 0147bc93-3b84-4b49-b191-d50ea87c9eb5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
72242025-04-30T06:30:07.294ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0147bc93-3b84-4b49-b191-d50ea87c9eb5", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
72252025-04-30T06:30:07.295ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
72262025-04-30T06:30:07.295ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
72272025-04-30T06:30:07.295ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
72282025-04-30T06:30:07.297ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
72292025-04-30T06:30:07.298ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72302025-04-30T06:30:07.298ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72312025-04-30T06:30:07.298ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72322025-04-30T06:30:07.298ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72332025-04-30T06:30:07.302ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72342025-04-30T06:30:07.308ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
72352025-04-30T06:30:07.309ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72362025-04-30T06:30:07.309ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72372025-04-30T06:30:07.309ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72382025-04-30T06:30:07.378ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
72392025-04-30T06:30:07.389ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
72402025-04-30T06:30:07.390ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
72412025-04-30T06:30:07.390ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 556.144151ms iteration = 22 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
72422025-04-30T06:30:07.390ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 23 reason = Signaled
72432025-04-30T06:30:07.390ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
72442025-04-30T06:30:07.390ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72452025-04-30T06:30:07.390ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72462025-04-30T06:30:07.390ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72472025-04-30T06:30:07.391ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72482025-04-30T06:30:07.401ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72492025-04-30T06:30:07.410ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
72502025-04-30T06:30:07.411ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72512025-04-30T06:30:07.411ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72522025-04-30T06:30:07.412ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72532025-04-30T06:30:07.416ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
72542025-04-30T06:30:07.416ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
72552025-04-30T06:30:07.426ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 08 1f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51773 req_id = c4987269-67b2-4c4e-9c9e-c9ccbd64621b
72562025-04-30T06:30:07.426ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2079,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51773 req_id = c4987269-67b2-4c4e-9c9e-c9ccbd64621b
72572025-04-30T06:30:07.426ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
72582025-04-30T06:30:07.427ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
72592025-04-30T06:30:07.427ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51773 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c4987269-67b2-4c4e-9c9e-c9ccbd64621b
72602025-04-30T06:30:07.427ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3472, tv_nsec: 226396131 } })
72612025-04-30T06:30:07.427ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
72622025-04-30T06:30:07.436ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72632025-04-30T06:30:07.438ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72642025-04-30T06:30:07.438ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72652025-04-30T06:30:07.438ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
72662025-04-30T06:30:07.438ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72672025-04-30T06:30:07.441ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8d 56 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .V..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53999 req_id = 7e289e04-025a-404c-ba63-43b24d74decb
72682025-04-30T06:30:07.441ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36182,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53999 req_id = 7e289e04-025a-404c-ba63-43b24d74decb
72692025-04-30T06:30:07.441ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
72702025-04-30T06:30:07.444ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53999 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7e289e04-025a-404c-ba63-43b24d74decb
72712025-04-30T06:30:07.538ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
72722025-04-30T06:30:07.538ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72732025-04-30T06:30:07.547ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72742025-04-30T06:30:07.547ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72752025-04-30T06:30:07.547ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
72762025-04-30T06:30:07.548ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72772025-04-30T06:30:07.548ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72782025-04-30T06:30:07.548ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72792025-04-30T06:30:07.549ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: bc 1c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61944 req_id = db1ff604-a57f-4546-920e-fef4eb7f69da
72802025-04-30T06:30:07.549ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48156,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61944 req_id = db1ff604-a57f-4546-920e-fef4eb7f69da
72812025-04-30T06:30:07.549ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
72822025-04-30T06:30:07.549ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48156, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61944 req_id = db1ff604-a57f-4546-920e-fef4eb7f69da
72832025-04-30T06:30:07.555ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
72842025-04-30T06:30:07.555ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
72852025-04-30T06:30:07.555ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
72862025-04-30T06:30:07.555ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72872025-04-30T06:30:07.556ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72882025-04-30T06:30:07.556ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72892025-04-30T06:30:07.556ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72902025-04-30T06:30:07.556ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72912025-04-30T06:30:07.556ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
72922025-04-30T06:30:07.557ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72932025-04-30T06:30:07.558ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0853195-44ff-401e-88df-4749377ec9ce", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
72942025-04-30T06:30:07.558ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
72952025-04-30T06:30:07.561ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "225be013-0a97-4256-8b49-180066fafbc1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
72962025-04-30T06:30:07.561ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
72972025-04-30T06:30:07.564ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4bf0f51-9988-4129-b78d-f3f0021745aa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
72982025-04-30T06:30:07.564ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
72992025-04-30T06:30:07.572ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78729514-e2cd-4ad3-90bd-8c9f6209f7cd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73002025-04-30T06:30:07.573ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
73012025-04-30T06:30:07.578ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27079558-f86f-43e7-9a3b-00c8f5b4e104", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73022025-04-30T06:30:07.578ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
73032025-04-30T06:30:07.594ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e20473ec-a01a-4b50-8545-6d2eed98ae98", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73042025-04-30T06:30:07.594ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
73052025-04-30T06:30:07.598ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
73062025-04-30T06:30:07.601ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "380cea0d-473b-411c-af75-454393cc9654", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73072025-04-30T06:30:07.601ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
73082025-04-30T06:30:07.603ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76b313bf-5bcb-4676-9f19-8aa151ecffd0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73092025-04-30T06:30:07.604ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
73102025-04-30T06:30:07.605ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77db9109-4537-4116-98eb-0537bf80114b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73112025-04-30T06:30:07.605ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
73122025-04-30T06:30:07.606ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43be1ef9-7e77-4e2e-9af8-e5d7776bf07d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73132025-04-30T06:30:07.607ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
73142025-04-30T06:30:07.611ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86f69590-35ff-4e09-8e12-9aee1253cde6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73152025-04-30T06:30:07.611ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
73162025-04-30T06:30:07.612ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9bcba06-cbb6-423c-9f26-2f8a6a118001", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73172025-04-30T06:30:07.612ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
73182025-04-30T06:30:07.614ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5270b1e2-cbbb-4322-b928-4e891a476bdf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73192025-04-30T06:30:07.614ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
73202025-04-30T06:30:07.619ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9442f1d4-5561-41ce-8f31-a62380aadac4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73212025-04-30T06:30:07.619ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
73222025-04-30T06:30:07.622ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ae0fccc-13ae-4811-8949-643483061a09", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73232025-04-30T06:30:07.622ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
73242025-04-30T06:30:07.627ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e1101f6-2130-49c0-99c8-1e07448b485a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73252025-04-30T06:30:07.627ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
73262025-04-30T06:30:07.634ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb0dd674-5689-4f83-9bfa-42bebbf01e92", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73272025-04-30T06:30:07.634ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
73282025-04-30T06:30:07.637ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef29baa8-1d15-4503-8a48-9370b0b7dea2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73292025-04-30T06:30:07.637ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
73302025-04-30T06:30:07.645ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01ed2fb8-fd14-4eb9-b81c-117dd134970b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73312025-04-30T06:30:07.645ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
73322025-04-30T06:30:07.647ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fe5eaed-8f3a-41a0-9439-3c307a280836", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73332025-04-30T06:30:07.648ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
73342025-04-30T06:30:07.648ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73352025-04-30T06:30:07.649ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73362025-04-30T06:30:07.649ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73372025-04-30T06:30:07.649ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
73382025-04-30T06:30:07.654ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73392025-04-30T06:30:07.659ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb8e0086-cb89-4a0f-ba51-68c8f8096de8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73402025-04-30T06:30:07.660ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
73412025-04-30T06:30:07.661ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73422025-04-30T06:30:07.671ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73432025-04-30T06:30:07.671ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73442025-04-30T06:30:07.671ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
73452025-04-30T06:30:07.672ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73462025-04-30T06:30:07.673ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39307de1-3e8f-45eb-b57b-2e852e6cc3e0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73472025-04-30T06:30:07.674ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
73482025-04-30T06:30:07.675ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73492025-04-30T06:30:07.675ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
73502025-04-30T06:30:07.676ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73512025-04-30T06:30:07.676ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "379b4867-9035-4a18-b9b9-f51b227c1073", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73522025-04-30T06:30:07.677ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
73532025-04-30T06:30:07.679ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8bc2424-4c1a-4bc3-b00d-4d1f528ce769", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73542025-04-30T06:30:07.680ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
73552025-04-30T06:30:07.685ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca6b9346-3dba-44cd-9799-4add437915bf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73562025-04-30T06:30:07.685ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
73572025-04-30T06:30:07.691ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9b800de-4f1e-4491-8f8e-b2dfea956255", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73582025-04-30T06:30:07.691ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
73592025-04-30T06:30:07.704ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "832dbcc0-ceed-4a92-bcda-23044e5d1d4c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73602025-04-30T06:30:07.704ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
73612025-04-30T06:30:07.707ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4118df22-0f64-491b-b2ec-031036f640d1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73622025-04-30T06:30:07.707ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
73632025-04-30T06:30:07.711ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
73642025-04-30T06:30:07.714ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7487f92-e9d8-477e-807f-97858e42b620", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73652025-04-30T06:30:07.714ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
73662025-04-30T06:30:07.718ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5a6d23d-2ca0-4097-9295-6803cd162c23", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73672025-04-30T06:30:07.718ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
73682025-04-30T06:30:07.720ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a65828e1-d29d-408f-9f01-2194ab2bea63", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73692025-04-30T06:30:07.720ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
73702025-04-30T06:30:07.721ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4360dc9-f5bf-4503-831b-fe521490a73c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73712025-04-30T06:30:07.721ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
73722025-04-30T06:30:07.723ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "415f66ca-5510-4270-bdec-1ca9b592f6cb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73732025-04-30T06:30:07.723ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
73742025-04-30T06:30:07.725ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9df6a59b-d8c3-4d93-bc4d-63b531803976", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73752025-04-30T06:30:07.725ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
73762025-04-30T06:30:07.727ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af439eba-4b67-4213-beae-bbe9bc9a66b9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73772025-04-30T06:30:07.727ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
73782025-04-30T06:30:07.735ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46e9914a-8674-4ed5-aae1-f95066143554", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73792025-04-30T06:30:07.736ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
73802025-04-30T06:30:07.737ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f05d7be7-bdc7-4af2-baa7-75dbcd90dc68", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73812025-04-30T06:30:07.738ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
73822025-04-30T06:30:07.741ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74a25eca-a863-44cc-a104-b5323cfdfc33", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73832025-04-30T06:30:07.741ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
73842025-04-30T06:30:07.743ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3eef407-158b-40f4-9682-00307f2acc23", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73852025-04-30T06:30:07.744ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
73862025-04-30T06:30:07.746ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c78018a6-f6c9-4092-8469-bbd720f3e557", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
73872025-04-30T06:30:07.746ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
73882025-04-30T06:30:07.753ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
73892025-04-30T06:30:07.755ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
73902025-04-30T06:30:07.759ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
73912025-04-30T06:30:07.762ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
73922025-04-30T06:30:07.764ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73932025-04-30T06:30:07.764ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73942025-04-30T06:30:07.764ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73952025-04-30T06:30:07.764ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
73962025-04-30T06:30:07.767ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73972025-04-30T06:30:07.772ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
73982025-04-30T06:30:07.772ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
73992025-04-30T06:30:07.779ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74002025-04-30T06:30:07.789ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74012025-04-30T06:30:07.789ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74022025-04-30T06:30:07.789ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
74032025-04-30T06:30:07.790ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74042025-04-30T06:30:07.791ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
74052025-04-30T06:30:07.791ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74062025-04-30T06:30:07.791ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ef419a1-3405-4654-89fd-92e7158ea82a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
74072025-04-30T06:30:07.791ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
74082025-04-30T06:30:07.795ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74092025-04-30T06:30:07.795ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
74102025-04-30T06:30:07.795ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74112025-04-30T06:30:07.795ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4747d8b0-49db-4fd6-bf59-e8111ba3f3ee", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
74122025-04-30T06:30:07.796ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
74132025-04-30T06:30:07.803ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82e8b391-18ea-472f-a70a-8b25aed4b20a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
74142025-04-30T06:30:07.804ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
74152025-04-30T06:30:07.808ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e56479b0-8f51-4b29-ba8c-7f1d8397f59f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
74162025-04-30T06:30:07.809ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
74172025-04-30T06:30:07.812ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa114e09-f2fb-47c2-827f-9c58ab38ccdf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
74182025-04-30T06:30:07.812ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
74192025-04-30T06:30:07.812ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74202025-04-30T06:30:07.812ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74212025-04-30T06:30:07.812ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74222025-04-30T06:30:07.820ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74232025-04-30T06:30:07.825ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74242025-04-30T06:30:07.831ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
74252025-04-30T06:30:07.831ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
74262025-04-30T06:30:07.831ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74272025-04-30T06:30:07.831ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74282025-04-30T06:30:07.831ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74292025-04-30T06:30:07.831ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74302025-04-30T06:30:07.837ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74312025-04-30T06:30:07.842ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
74322025-04-30T06:30:07.842ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74332025-04-30T06:30:07.842ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74342025-04-30T06:30:07.842ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74352025-04-30T06:30:07.930ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
74362025-04-30T06:30:07.931ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
74372025-04-30T06:30:07.932ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:47437 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
74382025-04-30T06:30:07.932ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
74392025-04-30T06:30:07.933ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:47437 req_id = 9c0cbf71-acfc-49be-934d-8d297ff0a027 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
74402025-04-30T06:30:07.936ZINFOtest_image_promotion (dropshot): request completed latency_us = 2407 local_addr = [::1]:60070 method = GET remote_addr = [::1]:47437 req_id = 9c0cbf71-acfc-49be-934d-8d297ff0a027 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
74412025-04-30T06:30:07.937ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c0cbf71-acfc-49be-934d-8d297ff0a027", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:06 GMT"} })
74422025-04-30T06:30:07.940ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
74432025-04-30T06:30:07.940ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
74442025-04-30T06:30:07.940ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
74452025-04-30T06:30:07.943ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
74462025-04-30T06:30:07.944ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74472025-04-30T06:30:07.944ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74482025-04-30T06:30:07.944ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74492025-04-30T06:30:07.946ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74502025-04-30T06:30:07.947ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
74512025-04-30T06:30:07.953ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74522025-04-30T06:30:07.954ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
74532025-04-30T06:30:07.955ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74542025-04-30T06:30:07.955ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74552025-04-30T06:30:07.956ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74562025-04-30T06:30:08.004ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74572025-04-30T06:30:08.004ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74582025-04-30T06:30:08.005ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74592025-04-30T06:30:08.005ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
74602025-04-30T06:30:08.006ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74612025-04-30T06:30:08.018ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74622025-04-30T06:30:08.025ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74632025-04-30T06:30:08.026ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74642025-04-30T06:30:08.026ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
74652025-04-30T06:30:08.026ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74662025-04-30T06:30:08.037ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
74672025-04-30T06:30:08.037ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
74682025-04-30T06:30:08.038ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 647.703117ms iteration = 23 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
74692025-04-30T06:30:08.038ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 24 reason = Signaled
74702025-04-30T06:30:08.038ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
74712025-04-30T06:30:08.038ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74722025-04-30T06:30:08.038ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74732025-04-30T06:30:08.038ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74742025-04-30T06:30:08.038ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74752025-04-30T06:30:08.039ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74762025-04-30T06:30:08.039ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
74772025-04-30T06:30:08.040ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74782025-04-30T06:30:08.041ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74792025-04-30T06:30:08.046ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
74802025-04-30T06:30:08.046ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74812025-04-30T06:30:08.046ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74822025-04-30T06:30:08.046ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74832025-04-30T06:30:08.053ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
74842025-04-30T06:30:08.054ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
74852025-04-30T06:30:08.055ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e3 2f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ./..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51018 req_id = dfb3fdff-52fd-49cd-b8e0-4e6b7956d069
74862025-04-30T06:30:08.055ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58159,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51018 req_id = dfb3fdff-52fd-49cd-b8e0-4e6b7956d069
74872025-04-30T06:30:08.056ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
74882025-04-30T06:30:08.056ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
74892025-04-30T06:30:08.057ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51018 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = dfb3fdff-52fd-49cd-b8e0-4e6b7956d069
74902025-04-30T06:30:08.058ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3472, tv_nsec: 857145738 } })
74912025-04-30T06:30:08.058ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
74922025-04-30T06:30:08.058ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 13 0e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54905 req_id = 30f7171b-e31a-4191-975c-61e5234f41c8
74932025-04-30T06:30:08.059ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4878,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54905 req_id = 30f7171b-e31a-4191-975c-61e5234f41c8
74942025-04-30T06:30:08.059ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
74952025-04-30T06:30:08.059ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54905 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 30f7171b-e31a-4191-975c-61e5234f41c8
74962025-04-30T06:30:08.141ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
74972025-04-30T06:30:08.143ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: fa 49 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .I..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50832 req_id = eae105df-9614-4ed2-bb0d-0bc975f4aedc
74982025-04-30T06:30:08.143ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64073,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50832 req_id = eae105df-9614-4ed2-bb0d-0bc975f4aedc
74992025-04-30T06:30:08.143ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
75002025-04-30T06:30:08.143ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64073, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50832 req_id = eae105df-9614-4ed2-bb0d-0bc975f4aedc
75012025-04-30T06:30:08.144ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
75022025-04-30T06:30:08.144ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
75032025-04-30T06:30:08.144ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
75042025-04-30T06:30:08.151ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "01c38daa-7df6-48a7-a85c-f47894cfed3d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75052025-04-30T06:30:08.151ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
75062025-04-30T06:30:08.164ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39fa06a9-1cd2-499f-9bbe-323b5c7e9a0f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75072025-04-30T06:30:08.164ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
75082025-04-30T06:30:08.172ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
75092025-04-30T06:30:08.174ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a65c77a-e5cd-4652-98a5-c91ecaded068", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75102025-04-30T06:30:08.174ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
75112025-04-30T06:30:08.180ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc2713c4-6389-43bf-89b6-efccc6927c51", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75122025-04-30T06:30:08.180ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
75132025-04-30T06:30:08.185ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "854381b8-0514-4a54-abc6-214405488fac", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75142025-04-30T06:30:08.185ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
75152025-04-30T06:30:08.187ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f3d5c17-1b8d-4130-af58-54125ba12099", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75162025-04-30T06:30:08.187ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
75172025-04-30T06:30:08.189ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "667e981a-1ba5-4d9a-affc-9c6be210a7e2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75182025-04-30T06:30:08.189ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
75192025-04-30T06:30:08.204ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25fa6135-3b8e-4642-b2f2-2ffad2f62111", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75202025-04-30T06:30:08.204ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
75212025-04-30T06:30:08.206ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69a16734-3f33-46c3-a1a1-3a729e73bcc7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75222025-04-30T06:30:08.206ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
75232025-04-30T06:30:08.212ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fa63b47-cebc-4c6e-9603-41d1b7230f83", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75242025-04-30T06:30:08.212ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
75252025-04-30T06:30:08.213ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ce4ea1c-e0d4-46c3-b8e0-8cfd5dd8f09d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75262025-04-30T06:30:08.213ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
75272025-04-30T06:30:08.214ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b63f2a2-0eb8-447a-a509-40f52a6b7745", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75282025-04-30T06:30:08.214ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
75292025-04-30T06:30:08.215ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "498a4568-cc45-4738-aa3d-da4a3e881ca1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75302025-04-30T06:30:08.215ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
75312025-04-30T06:30:08.217ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3071e9a0-fab5-44b5-8cad-754e036e12aa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75322025-04-30T06:30:08.217ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
75332025-04-30T06:30:08.220ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ac49af5-8d5c-4af5-bec1-57d1ba96d25e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75342025-04-30T06:30:08.221ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
75352025-04-30T06:30:08.222ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77b4021c-1041-4ae0-a665-15d7bd80c555", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75362025-04-30T06:30:08.223ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
75372025-04-30T06:30:08.223ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75382025-04-30T06:30:08.223ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75392025-04-30T06:30:08.223ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75402025-04-30T06:30:08.223ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
75412025-04-30T06:30:08.224ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75422025-04-30T06:30:08.227ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75432025-04-30T06:30:08.235ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75442025-04-30T06:30:08.235ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75452025-04-30T06:30:08.236ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
75462025-04-30T06:30:08.236ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75472025-04-30T06:30:08.238ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a8fe499-9a7e-43e3-bf1d-adcfb830f1a8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75482025-04-30T06:30:08.238ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
75492025-04-30T06:30:08.238ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75502025-04-30T06:30:08.239ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
75512025-04-30T06:30:08.240ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75522025-04-30T06:30:08.242ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "010f59e9-f2f3-4751-8447-3e5b3e57d2e7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75532025-04-30T06:30:08.243ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
75542025-04-30T06:30:08.248ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e09001a-ec51-4087-809c-b008981c7326", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75552025-04-30T06:30:08.248ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
75562025-04-30T06:30:08.249ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5e192f1-59b5-43ff-aa69-0fbe716f8a70", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75572025-04-30T06:30:08.249ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
75582025-04-30T06:30:08.251ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ee47f89-2ae5-4f8b-9499-1607394110e1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75592025-04-30T06:30:08.252ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
75602025-04-30T06:30:08.255ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "374c70b6-a5b6-4760-adec-c6d36abbcc24", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75612025-04-30T06:30:08.256ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
75622025-04-30T06:30:08.257ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ef82ece-c7b9-4236-9bba-c0645d4c52e4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75632025-04-30T06:30:08.257ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
75642025-04-30T06:30:08.264ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09bd3bad-0d87-48b6-9283-a96fc1488d7f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75652025-04-30T06:30:08.266ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
75662025-04-30T06:30:08.282ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e474c485-9ae2-4bd8-b265-9a07455ad3f7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75672025-04-30T06:30:08.289ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
75682025-04-30T06:30:08.306ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
75692025-04-30T06:30:08.326ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d42f85fe-fb03-4c12-80cc-b0c2df4cae28", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75702025-04-30T06:30:08.327ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
75712025-04-30T06:30:08.328ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5ceb8b3-5eab-4b9f-bb01-fa4670fb9c1c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75722025-04-30T06:30:08.329ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
75732025-04-30T06:30:08.330ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa5ba4e5-6eb7-4d88-9130-864894db7304", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75742025-04-30T06:30:08.330ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
75752025-04-30T06:30:08.333ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d9ba4e6-7986-485d-98c9-7fbc31d9a228", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75762025-04-30T06:30:08.333ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
75772025-04-30T06:30:08.335ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36437790-7a44-46a6-8753-3af2da6dd455", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75782025-04-30T06:30:08.336ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
75792025-04-30T06:30:08.337ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f31305a0-bb21-465b-9378-35cbb08de4c1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75802025-04-30T06:30:08.338ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
75812025-04-30T06:30:08.339ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cd129e2-125e-4439-a0dc-227d06e73e69", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75822025-04-30T06:30:08.340ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
75832025-04-30T06:30:08.341ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66b5252b-6482-4433-83c4-bcc258be7197", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75842025-04-30T06:30:08.341ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
75852025-04-30T06:30:08.342ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "471341a8-8e71-48bb-98c9-f142a54d27ec", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75862025-04-30T06:30:08.342ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
75872025-04-30T06:30:08.344ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd72f847-abc4-47f7-951a-3eb3223f05a7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75882025-04-30T06:30:08.344ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
75892025-04-30T06:30:08.346ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "434bc50c-eb4c-4e70-93a0-db03717b5363", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75902025-04-30T06:30:08.346ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
75912025-04-30T06:30:08.349ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99994696-c364-4aa0-ace1-49af44798865", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75922025-04-30T06:30:08.350ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
75932025-04-30T06:30:08.351ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e2c3bb3-6ba9-47db-b8c9-58d453778944", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75942025-04-30T06:30:08.352ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
75952025-04-30T06:30:08.355ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bac2313b-0bd3-4802-8c1c-1b9f1225c028", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
75962025-04-30T06:30:08.356ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
75972025-04-30T06:30:08.358ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75982025-04-30T06:30:08.359ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75992025-04-30T06:30:08.359ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76002025-04-30T06:30:08.359ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
76012025-04-30T06:30:08.360ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76022025-04-30T06:30:08.362ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76032025-04-30T06:30:08.372ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76042025-04-30T06:30:08.372ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76052025-04-30T06:30:08.372ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
76062025-04-30T06:30:08.373ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76072025-04-30T06:30:08.375ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "230afa47-b603-41cc-ba34-c9b59be4dc1a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
76082025-04-30T06:30:08.376ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
76092025-04-30T06:30:08.377ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76102025-04-30T06:30:08.377ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
76112025-04-30T06:30:08.378ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76122025-04-30T06:30:08.388ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d1a514e-96a7-419e-8be2-bb0dc80bf41e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
76132025-04-30T06:30:08.389ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
76142025-04-30T06:30:08.390ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "318b275d-d117-4db9-b468-5f0f5061da55", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
76152025-04-30T06:30:08.390ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
76162025-04-30T06:30:08.393ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "916a74af-556b-4388-9911-a8e83d1dd5da", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
76172025-04-30T06:30:08.394ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
76182025-04-30T06:30:08.395ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ea09842-f447-4c95-ae4d-2b1689bab5e4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
76192025-04-30T06:30:08.399ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
76202025-04-30T06:30:08.403ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87dd8089-cc76-47a0-a061-159939158f56", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
76212025-04-30T06:30:08.403ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
76222025-04-30T06:30:08.403ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76232025-04-30T06:30:08.403ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76242025-04-30T06:30:08.403ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76252025-04-30T06:30:08.404ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76262025-04-30T06:30:08.419ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76272025-04-30T06:30:08.425ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
76282025-04-30T06:30:08.425ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
76292025-04-30T06:30:08.425ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76302025-04-30T06:30:08.425ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76312025-04-30T06:30:08.425ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76322025-04-30T06:30:08.426ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76332025-04-30T06:30:08.428ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76342025-04-30T06:30:08.434ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
76352025-04-30T06:30:08.434ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76362025-04-30T06:30:08.434ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76372025-04-30T06:30:08.435ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76382025-04-30T06:30:08.523ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
76392025-04-30T06:30:08.527ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
76402025-04-30T06:30:08.528ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:57832 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
76412025-04-30T06:30:08.528ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
76422025-04-30T06:30:08.528ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76432025-04-30T06:30:08.530ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 1b581daa-3f81-40d2-a11b-11a19394a8c4 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76442025-04-30T06:30:08.530ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:57832 req_id = c3e33b3f-2b73-479a-ac61-b5cd0ae03c2e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
76452025-04-30T06:30:08.530ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 1b581daa-3f81-40d2-a11b-11a19394a8c4 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76462025-04-30T06:30:08.530ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 1b581daa-3f81-40d2-a11b-11a19394a8c4 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76472025-04-30T06:30:08.531ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 1b581daa-3f81-40d2-a11b-11a19394a8c4 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76482025-04-30T06:30:08.532ZINFOtest_image_promotion (dropshot): request completed latency_us = 1648 local_addr = [::1]:60070 method = GET remote_addr = [::1]:57832 req_id = c3e33b3f-2b73-479a-ac61-b5cd0ae03c2e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
76492025-04-30T06:30:08.534ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3e33b3f-2b73-479a-ac61-b5cd0ae03c2e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
76502025-04-30T06:30:08.538ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
76512025-04-30T06:30:08.538ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
76522025-04-30T06:30:08.538ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
76532025-04-30T06:30:08.541ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
76542025-04-30T06:30:08.541ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76552025-04-30T06:30:08.541ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76562025-04-30T06:30:08.541ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76572025-04-30T06:30:08.542ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76582025-04-30T06:30:08.542ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 12934 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 1b581daa-3f81-40d2-a11b-11a19394a8c4 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76592025-04-30T06:30:08.549ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76602025-04-30T06:30:08.549ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b581daa-3f81-40d2-a11b-11a19394a8c4", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
76612025-04-30T06:30:08.551ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
76622025-04-30T06:30:08.551ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76632025-04-30T06:30:08.553ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
76642025-04-30T06:30:08.553ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = cd9233e3-6c75-4e09-b32b-fd4cab6d8bb6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
76652025-04-30T06:30:08.555ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = cd9233e3-6c75-4e09-b32b-fd4cab6d8bb6 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
76662025-04-30T06:30:08.555ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = cd9233e3-6c75-4e09-b32b-fd4cab6d8bb6 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
76672025-04-30T06:30:08.555ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = cd9233e3-6c75-4e09-b32b-fd4cab6d8bb6 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
76682025-04-30T06:30:08.557ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76692025-04-30T06:30:08.561ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
76702025-04-30T06:30:08.561ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76712025-04-30T06:30:08.561ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76722025-04-30T06:30:08.562ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76732025-04-30T06:30:08.568ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 14128 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = cd9233e3-6c75-4e09-b32b-fd4cab6d8bb6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
76742025-04-30T06:30:08.568ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd9233e3-6c75-4e09-b32b-fd4cab6d8bb6", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
76752025-04-30T06:30:08.568ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76762025-04-30T06:30:08.568ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
76772025-04-30T06:30:08.568ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
76782025-04-30T06:30:08.569ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76792025-04-30T06:30:08.570ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76802025-04-30T06:30:08.570ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76812025-04-30T06:30:08.570ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76822025-04-30T06:30:08.598ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
76832025-04-30T06:30:08.598ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
76842025-04-30T06:30:08.598ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 560.425371ms iteration = 24 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
76852025-04-30T06:30:08.598ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 25 reason = Signaled
76862025-04-30T06:30:08.598ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
76872025-04-30T06:30:08.598ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76882025-04-30T06:30:08.598ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76892025-04-30T06:30:08.598ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76902025-04-30T06:30:08.599ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76912025-04-30T06:30:08.603ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76922025-04-30T06:30:08.608ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
76932025-04-30T06:30:08.608ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76942025-04-30T06:30:08.608ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76952025-04-30T06:30:08.608ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76962025-04-30T06:30:08.609ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76972025-04-30T06:30:08.609ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76982025-04-30T06:30:08.609ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76992025-04-30T06:30:08.609ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
77002025-04-30T06:30:08.609ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77012025-04-30T06:30:08.611ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
77022025-04-30T06:30:08.611ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
77032025-04-30T06:30:08.613ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77042025-04-30T06:30:08.620ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77052025-04-30T06:30:08.620ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77062025-04-30T06:30:08.620ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
77072025-04-30T06:30:08.620ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77082025-04-30T06:30:08.620ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cc c3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55037 req_id = 20c2a397-524c-4eee-b51c-df12f8a87065
77092025-04-30T06:30:08.620ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52419,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55037 req_id = 20c2a397-524c-4eee-b51c-df12f8a87065
77102025-04-30T06:30:08.620ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
77112025-04-30T06:30:08.621ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
77122025-04-30T06:30:08.621ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55037 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 20c2a397-524c-4eee-b51c-df12f8a87065
77132025-04-30T06:30:08.621ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3473, tv_nsec: 420568662 } })
77142025-04-30T06:30:08.621ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
77152025-04-30T06:30:08.622ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 31 bf 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 1...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62216 req_id = 6fcf18bb-5e95-4f2e-ad7f-d92ce2615f71
77162025-04-30T06:30:08.622ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12735,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62216 req_id = 6fcf18bb-5e95-4f2e-ad7f-d92ce2615f71
77172025-04-30T06:30:08.622ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
77182025-04-30T06:30:08.622ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62216 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6fcf18bb-5e95-4f2e-ad7f-d92ce2615f71
77192025-04-30T06:30:08.681ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
77202025-04-30T06:30:08.681ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77212025-04-30T06:30:08.681ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
77222025-04-30T06:30:08.682ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77232025-04-30T06:30:08.682ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
77242025-04-30T06:30:08.682ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
77252025-04-30T06:30:08.683ZINFOtest_image_promotion (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = 127.0.0.1:60931 remote_addr = 127.0.0.1:53785
77262025-04-30T06:30:08.683ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: bc 76 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .v..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51576 req_id = 637686b9-2c32-428a-a58a-c1fe5cd4ec92
77272025-04-30T06:30:08.683ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48246,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51576 req_id = 637686b9-2c32-428a-a58a-c1fe5cd4ec92
77282025-04-30T06:30:08.683ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
77292025-04-30T06:30:08.683ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48246, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51576 req_id = 637686b9-2c32-428a-a58a-c1fe5cd4ec92
77302025-04-30T06:30:08.684ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
77312025-04-30T06:30:08.684ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
77322025-04-30T06:30:08.684ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
77332025-04-30T06:30:08.684ZTRACtest_image_promotion (dropshot): incoming request local_addr = 127.0.0.1:60931 method = GET remote_addr = 127.0.0.1:53785 req_id = 8a8825b1-5da4-4156-ac0d-7432f492fa50 uri = /d9b51ab0-9dce-44da-9764-8fd37f3b0498
77342025-04-30T06:30:08.690ZINFOtest_image_promotion (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 6027 local_addr = 127.0.0.1:60931 method = GET remote_addr = 127.0.0.1:53785 req_id = 8a8825b1-5da4-4156-ac0d-7432f492fa50 response_code = 200 uri = /d9b51ab0-9dce-44da-9764-8fd37f3b0498
77352025-04-30T06:30:08.691ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
77362025-04-30T06:30:08.711ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
77372025-04-30T06:30:08.711ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
77382025-04-30T06:30:08.711ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
77392025-04-30T06:30:08.711ZDEBGtest_image_promotion (results-sink): inserting 18 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77402025-04-30T06:30:08.711ZDEBGtest_image_promotion (results-sink-cluster): inserting 18 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77412025-04-30T06:30:08.712ZDEBGtest_image_promotion (clickhouse-client): unrolling 18 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
77422025-04-30T06:30:08.713ZDEBGtest_image_promotion (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
77432025-04-30T06:30:08.713ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('integration_target:integration_metric') FORMAT Native;
77442025-04-30T06:30:08.715ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "84d7b1f7-a7a3-4cce-ada9-1184c5965ff2", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77452025-04-30T06:30:08.715ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
77462025-04-30T06:30:08.719ZTRACtest_image_promotion (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
77472025-04-30T06:30:08.719ZDEBGtest_image_promotion (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:30:08.713572889Z } timeseries_name = http_service:request_latency_histogram
77482025-04-30T06:30:08.729ZDEBGtest_image_promotion (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
77492025-04-30T06:30:08.729ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
77502025-04-30T06:30:08.734ZTRACtest_image_promotion (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
77512025-04-30T06:30:08.735ZDEBGtest_image_promotion (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:30:08.729653388Z } timeseries_name = collection_target:cpus_provisioned
77522025-04-30T06:30:08.736ZDEBGtest_image_promotion (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
77532025-04-30T06:30:08.736ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
77542025-04-30T06:30:08.738ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0442a9a9-d7d1-429e-8768-b603775bbb31", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77552025-04-30T06:30:08.740ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
77562025-04-30T06:30:08.741ZTRACtest_image_promotion (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
77572025-04-30T06:30:08.741ZDEBGtest_image_promotion (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:30:08.736184461Z } timeseries_name = collection_target:ram_provisioned
77582025-04-30T06:30:08.742ZDEBGtest_image_promotion (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
77592025-04-30T06:30:08.742ZTRACtest_image_promotion (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'collection_target:ram_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
77602025-04-30T06:30:08.745ZTRACtest_image_promotion (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
77612025-04-30T06:30:08.745ZDEBGtest_image_promotion (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:30:08.742430135Z } timeseries_name = collection_target:virtual_disk_space_provisioned
77622025-04-30T06:30:08.752ZDEBGtest_image_promotion (clickhouse-client): inserting 4 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
77632025-04-30T06:30:08.752ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 6 n_rows = 4 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
77642025-04-30T06:30:08.753ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aef952b0-0fdf-4810-bcea-34f80397286b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77652025-04-30T06:30:08.753ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
77662025-04-30T06:30:08.757ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
77672025-04-30T06:30:08.758ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dc8342d-3daa-42f4-bd32-0958cf8754ea", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77682025-04-30T06:30:08.758ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
77692025-04-30T06:30:08.762ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 12 table_name = fields_string
77702025-04-30T06:30:08.762ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
77712025-04-30T06:30:08.765ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 5 table_name = fields_u16
77722025-04-30T06:30:08.765ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 17 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
77732025-04-30T06:30:08.766ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19921def-9d96-4a6f-879f-5aefcfcf1f63", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77742025-04-30T06:30:08.766ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
77752025-04-30T06:30:08.769ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
77762025-04-30T06:30:08.772ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 17 table_name = fields_uuid
77772025-04-30T06:30:08.772ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
77782025-04-30T06:30:08.773ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8dae281-b905-41a0-802b-2152bc25df73", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77792025-04-30T06:30:08.773ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
77802025-04-30T06:30:08.776ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0e799f3-60c5-4f35-92e2-e21e84833fb2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77812025-04-30T06:30:08.776ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
77822025-04-30T06:30:08.777ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 5 table_name = measurements_histogramu64
77832025-04-30T06:30:08.777ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 13 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
77842025-04-30T06:30:08.780ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1490f0f8-c161-4dfb-8e0b-bdea49a4d82c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77852025-04-30T06:30:08.780ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
77862025-04-30T06:30:08.782ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f890937a-f0e0-44dd-9564-3bb2994c51bb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77872025-04-30T06:30:08.782ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
77882025-04-30T06:30:08.782ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bc4ee68-6059-41de-90de-b7dd45e88039", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77892025-04-30T06:30:08.783ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
77902025-04-30T06:30:08.788ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8a850b5-78a9-4fc5-b569-bed0f59d08b2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77912025-04-30T06:30:08.788ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
77922025-04-30T06:30:08.788ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 13 table_name = measurements_i64
77932025-04-30T06:30:08.788ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77942025-04-30T06:30:08.788ZTRACtest_image_promotion (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77952025-04-30T06:30:08.788ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "edc73d3d-c7e1-4105-b64f-1f4dc40d59e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77962025-04-30T06:30:08.788ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
77972025-04-30T06:30:08.788ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff651f7e-9bd8-4d00-828e-fe453e337b54", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
77982025-04-30T06:30:08.788ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
77992025-04-30T06:30:08.797ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1382e99b-7e13-4f87-b6c6-49fcb2424c5d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78002025-04-30T06:30:08.798ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
78012025-04-30T06:30:08.800ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d7d1aae-adac-41ac-83c4-6282eecab96a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78022025-04-30T06:30:08.800ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
78032025-04-30T06:30:08.806ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "35fcd2d9-7147-4838-bc75-3f1737e47f82", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78042025-04-30T06:30:08.806ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
78052025-04-30T06:30:08.811ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "319e0d37-7d67-4ccc-b910-da2ff4815da1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78062025-04-30T06:30:08.811ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
78072025-04-30T06:30:08.813ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
78082025-04-30T06:30:08.813ZTRACtest_image_promotion (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78092025-04-30T06:30:08.815ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51ec4192-cbc2-44f8-b580-c43cadd84bbc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78102025-04-30T06:30:08.815ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
78112025-04-30T06:30:08.817ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38b1873d-54f5-4e7c-831b-22b5f4832ad6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78122025-04-30T06:30:08.818ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
78132025-04-30T06:30:08.819ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e48405f9-5657-434a-b8c0-c625b9adbc66", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78142025-04-30T06:30:08.819ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
78152025-04-30T06:30:08.820ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38060c13-41fd-4437-bdd6-16afc146c30d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78162025-04-30T06:30:08.820ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
78172025-04-30T06:30:08.824ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78182025-04-30T06:30:08.825ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78192025-04-30T06:30:08.825ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78202025-04-30T06:30:08.825ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
78212025-04-30T06:30:08.829ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78222025-04-30T06:30:08.832ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20a2aeef-27b1-4451-b059-896d8dbbd819", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78232025-04-30T06:30:08.833ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
78242025-04-30T06:30:08.833ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78252025-04-30T06:30:08.848ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78262025-04-30T06:30:08.848ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78272025-04-30T06:30:08.848ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
78282025-04-30T06:30:08.849ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78292025-04-30T06:30:08.852ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78302025-04-30T06:30:08.853ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
78312025-04-30T06:30:08.854ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78322025-04-30T06:30:08.855ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63028fad-3719-47a7-95ec-5d92d8abb994", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78332025-04-30T06:30:08.855ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
78342025-04-30T06:30:08.857ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34207cba-ab49-4e3b-90ac-228a665a59a7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78352025-04-30T06:30:08.858ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
78362025-04-30T06:30:08.864ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce7c10c7-ea46-49ba-86e7-9c93fc269713", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78372025-04-30T06:30:08.865ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
78382025-04-30T06:30:08.870ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac8c900d-8378-445b-af16-04419e774f92", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78392025-04-30T06:30:08.870ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
78402025-04-30T06:30:08.877ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "795f64b7-1f4b-4d52-ac3e-36159d59eeb1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78412025-04-30T06:30:08.878ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
78422025-04-30T06:30:08.887ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "275b3817-6e75-40d6-bcc0-7b9b52747d09", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78432025-04-30T06:30:08.888ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
78442025-04-30T06:30:08.891ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
78452025-04-30T06:30:08.893ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5950e75-047f-433e-a2ac-0649a7bb3731", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78462025-04-30T06:30:08.893ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
78472025-04-30T06:30:08.896ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "19be2bea-3705-41fc-b534-ed45dc6978b4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78482025-04-30T06:30:08.896ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
78492025-04-30T06:30:08.897ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "584701ea-8f55-4b60-808d-9ef5880d3344", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78502025-04-30T06:30:08.897ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
78512025-04-30T06:30:08.898ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27982d7d-e90f-4891-abb0-8a9dfb2fea2e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78522025-04-30T06:30:08.898ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
78532025-04-30T06:30:08.900ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c762f28-9af1-4b3b-ab90-28db7fce1a4e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78542025-04-30T06:30:08.900ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
78552025-04-30T06:30:08.904ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f26b1d4-8509-46aa-9b35-1f130affa8f4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78562025-04-30T06:30:08.904ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
78572025-04-30T06:30:08.906ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e82a8bda-c26d-49a3-bec1-e8875bf65536", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78582025-04-30T06:30:08.906ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
78592025-04-30T06:30:08.909ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "765c196d-7235-4dce-8e2b-1f927fab5c20", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78602025-04-30T06:30:08.909ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
78612025-04-30T06:30:08.915ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28b0817d-36e0-4759-9cf1-9f6a1b8b52b7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78622025-04-30T06:30:08.916ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
78632025-04-30T06:30:08.918ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2429ea30-98cb-4a65-8171-250b6a16786a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78642025-04-30T06:30:08.918ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
78652025-04-30T06:30:08.923ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ecd3f34-0961-44c4-90ed-48b006876a00", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78662025-04-30T06:30:08.923ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
78672025-04-30T06:30:08.928ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0ba16bb-71f2-4c8e-a839-a0640dc1ce70", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78682025-04-30T06:30:08.928ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
78692025-04-30T06:30:08.932ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbce1ad1-c98a-4aed-b240-2c0713bbd664", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78702025-04-30T06:30:08.933ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
78712025-04-30T06:30:08.936ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70f8d7f5-2e6a-466a-982a-9bc16c5daa49", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78722025-04-30T06:30:08.937ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
78732025-04-30T06:30:08.938ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8a7530b-f22c-48a6-876d-0e6a15b7aab3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78742025-04-30T06:30:08.938ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
78752025-04-30T06:30:08.939ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27f63731-1ae7-4c65-af27-d006549ad460", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78762025-04-30T06:30:08.939ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
78772025-04-30T06:30:08.941ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f3ab7f2-0f76-4d1b-8739-42a5c3fad266", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:07 GMT"} })
78782025-04-30T06:30:08.941ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
78792025-04-30T06:30:08.941ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78802025-04-30T06:30:08.941ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78812025-04-30T06:30:08.941ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78822025-04-30T06:30:08.943ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78832025-04-30T06:30:08.945ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78842025-04-30T06:30:08.945ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78852025-04-30T06:30:08.945ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78862025-04-30T06:30:08.945ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
78872025-04-30T06:30:08.948ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78882025-04-30T06:30:08.952ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78892025-04-30T06:30:08.960ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
78902025-04-30T06:30:08.960ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
78912025-04-30T06:30:08.960ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78922025-04-30T06:30:08.960ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78932025-04-30T06:30:08.960ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78942025-04-30T06:30:08.961ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78952025-04-30T06:30:08.963ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78962025-04-30T06:30:08.976ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78972025-04-30T06:30:08.976ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78982025-04-30T06:30:08.976ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
78992025-04-30T06:30:08.978ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79002025-04-30T06:30:08.982ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79012025-04-30T06:30:08.987ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79022025-04-30T06:30:08.988ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79032025-04-30T06:30:08.990ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79042025-04-30T06:30:08.991ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79052025-04-30T06:30:08.993ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79062025-04-30T06:30:08.993ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
79072025-04-30T06:30:08.994ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79082025-04-30T06:30:09.061ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
79092025-04-30T06:30:09.061ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
79102025-04-30T06:30:09.062ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:51138 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
79112025-04-30T06:30:09.063ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:51138 req_id = c9e565f7-30a7-4227-b147-d832fd1d4f1e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
79122025-04-30T06:30:09.066ZINFOtest_image_promotion (dropshot): request completed latency_us = 3382 local_addr = [::1]:60070 method = GET remote_addr = [::1]:51138 req_id = c9e565f7-30a7-4227-b147-d832fd1d4f1e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
79132025-04-30T06:30:09.067ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9e565f7-30a7-4227-b147-d832fd1d4f1e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
79142025-04-30T06:30:09.069ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
79152025-04-30T06:30:09.069ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
79162025-04-30T06:30:09.069ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
79172025-04-30T06:30:09.072ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
79182025-04-30T06:30:09.072ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79192025-04-30T06:30:09.072ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79202025-04-30T06:30:09.072ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79212025-04-30T06:30:09.073ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79222025-04-30T06:30:09.081ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79232025-04-30T06:30:09.089ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79242025-04-30T06:30:09.089ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79252025-04-30T06:30:09.089ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79262025-04-30T06:30:09.090ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79272025-04-30T06:30:09.168ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
79282025-04-30T06:30:09.170ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
79292025-04-30T06:30:09.171ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
79302025-04-30T06:30:09.171ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 572.832632ms iteration = 25 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
79312025-04-30T06:30:09.171ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 26 reason = Signaled
79322025-04-30T06:30:09.171ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
79332025-04-30T06:30:09.171ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79342025-04-30T06:30:09.171ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79352025-04-30T06:30:09.171ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79362025-04-30T06:30:09.175ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79372025-04-30T06:30:09.178ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79382025-04-30T06:30:09.186ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79392025-04-30T06:30:09.187ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79402025-04-30T06:30:09.187ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79412025-04-30T06:30:09.187ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79422025-04-30T06:30:09.192ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
79432025-04-30T06:30:09.192ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
79442025-04-30T06:30:09.193ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 44 bb 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 D............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63738 req_id = 5922f07f-d8c1-454f-8bc9-0faa90fc31d6
79452025-04-30T06:30:09.193ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17595,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63738 req_id = 5922f07f-d8c1-454f-8bc9-0faa90fc31d6
79462025-04-30T06:30:09.193ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
79472025-04-30T06:30:09.195ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
79482025-04-30T06:30:09.195ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63738 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5922f07f-d8c1-454f-8bc9-0faa90fc31d6
79492025-04-30T06:30:09.196ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3473, tv_nsec: 994568679 } })
79502025-04-30T06:30:09.196ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
79512025-04-30T06:30:09.197ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6d 92 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e m...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52895 req_id = bbed789b-3240-48ff-9a56-277888dc8dae
79522025-04-30T06:30:09.197ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28050,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52895 req_id = bbed789b-3240-48ff-9a56-277888dc8dae
79532025-04-30T06:30:09.198ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
79542025-04-30T06:30:09.198ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52895 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = bbed789b-3240-48ff-9a56-277888dc8dae
79552025-04-30T06:30:09.251ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
79562025-04-30T06:30:09.252ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79572025-04-30T06:30:09.252ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79582025-04-30T06:30:09.252ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79592025-04-30T06:30:09.252ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
79602025-04-30T06:30:09.256ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79612025-04-30T06:30:09.257ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 45 ed 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c E............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64528 req_id = 5e0d5e7f-633e-4d3f-9021-eadac0c3183f
79622025-04-30T06:30:09.257ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17901,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64528 req_id = 5e0d5e7f-633e-4d3f-9021-eadac0c3183f
79632025-04-30T06:30:09.257ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
79642025-04-30T06:30:09.258ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17901, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64528 req_id = 5e0d5e7f-633e-4d3f-9021-eadac0c3183f
79652025-04-30T06:30:09.260ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
79662025-04-30T06:30:09.260ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
79672025-04-30T06:30:09.260ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
79682025-04-30T06:30:09.261ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79692025-04-30T06:30:09.277ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79702025-04-30T06:30:09.277ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79712025-04-30T06:30:09.277ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
79722025-04-30T06:30:09.278ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79732025-04-30T06:30:09.279ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4d77fcc-e8e9-4ae2-9b6a-4409570274c7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
79742025-04-30T06:30:09.279ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
79752025-04-30T06:30:09.281ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79762025-04-30T06:30:09.281ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
79772025-04-30T06:30:09.282ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79782025-04-30T06:30:09.282ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34aacf22-0f33-44d6-bcd4-0dc6ca1bb177", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
79792025-04-30T06:30:09.283ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
79802025-04-30T06:30:09.289ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bcbe3b1-16cb-4dd3-b452-a4ff5d4e3164", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
79812025-04-30T06:30:09.289ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
79822025-04-30T06:30:09.293ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "482d962b-62f9-411a-868b-77a9f6ba4590", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
79832025-04-30T06:30:09.293ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
79842025-04-30T06:30:09.297ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3efa1e4e-884e-4f56-acb5-3bce54e5a24a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
79852025-04-30T06:30:09.297ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
79862025-04-30T06:30:09.300ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "181e6b49-0b19-4845-af8b-27a480d0c55e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
79872025-04-30T06:30:09.301ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
79882025-04-30T06:30:09.304ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c7f71cd-aa76-4545-89f0-a7014372879a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
79892025-04-30T06:30:09.305ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
79902025-04-30T06:30:09.317ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d19c724f-9359-4025-bc0f-91b1fd3c8cea", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
79912025-04-30T06:30:09.317ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
79922025-04-30T06:30:09.318ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d625c7cb-7c3f-4a73-b760-a931f98b4135", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
79932025-04-30T06:30:09.318ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
79942025-04-30T06:30:09.322ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "172a1033-3277-4a37-86d2-ecaa17dc4a13", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
79952025-04-30T06:30:09.323ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
79962025-04-30T06:30:09.323ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
79972025-04-30T06:30:09.329ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dee97497-3289-455e-a6df-c3c0f075f31f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
79982025-04-30T06:30:09.330ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
79992025-04-30T06:30:09.332ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dd63099-398a-4cd8-8186-050519b3c84c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80002025-04-30T06:30:09.332ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
80012025-04-30T06:30:09.333ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bf4ad96-2088-4c74-b7bb-f4afd1a709b7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80022025-04-30T06:30:09.334ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
80032025-04-30T06:30:09.338ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50b4832c-91da-4012-a320-44d18cf080ef", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80042025-04-30T06:30:09.338ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
80052025-04-30T06:30:09.341ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c5ee907-4a69-498c-80f2-a93db4518115", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80062025-04-30T06:30:09.341ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
80072025-04-30T06:30:09.344ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4035b50-3f2d-4935-97f2-3336eceae8ab", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80082025-04-30T06:30:09.344ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
80092025-04-30T06:30:09.346ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6acf4f53-7a28-4590-b62c-7bcd28bf3f1c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80102025-04-30T06:30:09.346ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
80112025-04-30T06:30:09.349ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "425426c0-03c4-4c8d-8ebd-2c281bd41e10", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80122025-04-30T06:30:09.349ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
80132025-04-30T06:30:09.353ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43e7b8ad-3057-4252-802d-9f2f3356c98b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80142025-04-30T06:30:09.353ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
80152025-04-30T06:30:09.354ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53603567-7419-4d34-84b3-5e45a311bca5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80162025-04-30T06:30:09.355ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
80172025-04-30T06:30:09.358ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c8b816f-9afb-4d5a-a0a0-858d900235a7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80182025-04-30T06:30:09.358ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
80192025-04-30T06:30:09.361ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b614038b-05e3-4a6f-bc58-d018fbb0bfcb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80202025-04-30T06:30:09.361ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
80212025-04-30T06:30:09.364ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "882ffee6-07ba-4449-a65e-41f0f0bf8aa2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80222025-04-30T06:30:09.364ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
80232025-04-30T06:30:09.367ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ba06ecb-0b5b-48d8-9aba-500f2bc4c7ef", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80242025-04-30T06:30:09.368ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
80252025-04-30T06:30:09.379ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80262025-04-30T06:30:09.379ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80272025-04-30T06:30:09.379ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80282025-04-30T06:30:09.379ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
80292025-04-30T06:30:09.381ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80302025-04-30T06:30:09.384ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80312025-04-30T06:30:09.395ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80322025-04-30T06:30:09.395ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80332025-04-30T06:30:09.395ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
80342025-04-30T06:30:09.397ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80352025-04-30T06:30:09.400ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c077d011-8d61-4901-b038-e72ae4ec54a1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80362025-04-30T06:30:09.402ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
80372025-04-30T06:30:09.402ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80382025-04-30T06:30:09.402ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
80392025-04-30T06:30:09.403ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80402025-04-30T06:30:09.409ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b43294df-ecc5-45f6-98dd-c139cb21a30a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80412025-04-30T06:30:09.409ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
80422025-04-30T06:30:09.413ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52c3db3c-4366-4f94-8352-cd0a8b54746e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80432025-04-30T06:30:09.413ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
80442025-04-30T06:30:09.443ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b37a7076-9e5c-4587-b5fa-f9ef30ac4d84", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80452025-04-30T06:30:09.444ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
80462025-04-30T06:30:09.448ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b8b492f-37ce-4185-8e7b-36eb99391087", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80472025-04-30T06:30:09.448ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
80482025-04-30T06:30:09.452ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4705d068-dbe6-4223-a851-33afb5e6db7c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80492025-04-30T06:30:09.453ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
80502025-04-30T06:30:09.454ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c847856-e1c1-4ac5-bca0-343ccaeca6af", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80512025-04-30T06:30:09.454ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
80522025-04-30T06:30:09.457ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dde22786-b355-41b8-81df-aed242e1810f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80532025-04-30T06:30:09.457ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
80542025-04-30T06:30:09.458ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5843a35-0986-4afd-986c-d2deae58a47a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80552025-04-30T06:30:09.458ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
80562025-04-30T06:30:09.463ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
80572025-04-30T06:30:09.465ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb59486d-ba40-45be-8eaf-d9f53dd6d736", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80582025-04-30T06:30:09.465ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
80592025-04-30T06:30:09.467ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d24983d9-b367-4da4-bd5b-a258f9073a0e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80602025-04-30T06:30:09.467ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
80612025-04-30T06:30:09.469ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2324b788-37f6-4364-bce1-6db37e9d1a56", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80622025-04-30T06:30:09.469ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
80632025-04-30T06:30:09.471ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16edeed0-577b-4a2d-8ae5-cc0fe7aa19d1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80642025-04-30T06:30:09.471ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
80652025-04-30T06:30:09.474ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa2f34dc-fe3b-4c90-8916-68c682baea3c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80662025-04-30T06:30:09.474ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
80672025-04-30T06:30:09.478ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57410307-2e52-4b97-9008-5e65fe062565", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80682025-04-30T06:30:09.478ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
80692025-04-30T06:30:09.487ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81eb3e82-3127-4c6a-9fc4-18c15a4ba4c2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80702025-04-30T06:30:09.487ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
80712025-04-30T06:30:09.491ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7abad899-abcb-4c8e-9886-3c267e48c3ea", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80722025-04-30T06:30:09.491ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
80732025-04-30T06:30:09.493ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cbed1e7-cd74-44c1-b7dd-488bae4633c1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80742025-04-30T06:30:09.494ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
80752025-04-30T06:30:09.496ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffe7bf9b-55ba-4678-b3d7-4df384c9ca2b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80762025-04-30T06:30:09.498ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
80772025-04-30T06:30:09.499ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b3d9390-b2ab-4f15-a97b-9164dcbbc6fd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80782025-04-30T06:30:09.499ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
80792025-04-30T06:30:09.501ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31ea86e1-0a54-4e3e-a5c8-c054ca00ff47", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
80802025-04-30T06:30:09.501ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
80812025-04-30T06:30:09.501ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80822025-04-30T06:30:09.501ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80832025-04-30T06:30:09.501ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80842025-04-30T06:30:09.502ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80852025-04-30T06:30:09.507ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80862025-04-30T06:30:09.514ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
80872025-04-30T06:30:09.514ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
80882025-04-30T06:30:09.514ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80892025-04-30T06:30:09.514ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80902025-04-30T06:30:09.514ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80912025-04-30T06:30:09.516ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80922025-04-30T06:30:09.518ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80932025-04-30T06:30:09.518ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80942025-04-30T06:30:09.518ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80952025-04-30T06:30:09.518ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
80962025-04-30T06:30:09.521ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80972025-04-30T06:30:09.524ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80982025-04-30T06:30:09.534ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
80992025-04-30T06:30:09.534ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81002025-04-30T06:30:09.534ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81012025-04-30T06:30:09.535ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81022025-04-30T06:30:09.537ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81032025-04-30T06:30:09.545ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81042025-04-30T06:30:09.546ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81052025-04-30T06:30:09.546ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
81062025-04-30T06:30:09.546ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81072025-04-30T06:30:09.603ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
81082025-04-30T06:30:09.603ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
81092025-04-30T06:30:09.603ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:57978 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
81102025-04-30T06:30:09.604ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81112025-04-30T06:30:09.604ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81122025-04-30T06:30:09.605ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:57978 req_id = dccf2a59-1a7b-4910-b3ec-634e017615ce sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81132025-04-30T06:30:09.607ZINFOtest_image_promotion (dropshot): request completed latency_us = 2607 local_addr = [::1]:60070 method = GET remote_addr = [::1]:57978 req_id = dccf2a59-1a7b-4910-b3ec-634e017615ce response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81142025-04-30T06:30:09.607ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81152025-04-30T06:30:09.608ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
81162025-04-30T06:30:09.609ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81172025-04-30T06:30:09.609ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81182025-04-30T06:30:09.610ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81192025-04-30T06:30:09.610ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81202025-04-30T06:30:09.610ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81212025-04-30T06:30:09.610ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "dccf2a59-1a7b-4910-b3ec-634e017615ce", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
81222025-04-30T06:30:09.612ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81232025-04-30T06:30:09.612ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81242025-04-30T06:30:09.612ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
81252025-04-30T06:30:09.614ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81262025-04-30T06:30:09.614ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81272025-04-30T06:30:09.614ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81282025-04-30T06:30:09.614ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81292025-04-30T06:30:09.615ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81302025-04-30T06:30:09.619ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81312025-04-30T06:30:09.625ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
81322025-04-30T06:30:09.625ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81332025-04-30T06:30:09.625ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81342025-04-30T06:30:09.626ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81352025-04-30T06:30:09.658ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
81362025-04-30T06:30:09.658ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
81372025-04-30T06:30:09.658ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 487.246748ms iteration = 26 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
81382025-04-30T06:30:09.658ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 27 reason = Signaled
81392025-04-30T06:30:09.658ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81402025-04-30T06:30:09.658ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81412025-04-30T06:30:09.658ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81422025-04-30T06:30:09.658ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81432025-04-30T06:30:09.659ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81442025-04-30T06:30:09.660ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
81452025-04-30T06:30:09.662ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81462025-04-30T06:30:09.666ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
81472025-04-30T06:30:09.666ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81482025-04-30T06:30:09.666ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81492025-04-30T06:30:09.667ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81502025-04-30T06:30:09.671ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
81512025-04-30T06:30:09.671ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
81522025-04-30T06:30:09.672ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: aa c8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49975 req_id = a5fda1f8-1cc6-4caa-b83b-032eb8702a13
81532025-04-30T06:30:09.672ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43720,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49975 req_id = a5fda1f8-1cc6-4caa-b83b-032eb8702a13
81542025-04-30T06:30:09.672ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
81552025-04-30T06:30:09.673ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
81562025-04-30T06:30:09.673ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49975 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a5fda1f8-1cc6-4caa-b83b-032eb8702a13
81572025-04-30T06:30:09.673ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3474, tv_nsec: 472421239 } })
81582025-04-30T06:30:09.673ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
81592025-04-30T06:30:09.674ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a4 2f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ./..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52845 req_id = 1af48e78-6ce8-4a21-be20-0e390a289e91
81602025-04-30T06:30:09.674ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42031,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52845 req_id = 1af48e78-6ce8-4a21-be20-0e390a289e91
81612025-04-30T06:30:09.674ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
81622025-04-30T06:30:09.674ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52845 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1af48e78-6ce8-4a21-be20-0e390a289e91
81632025-04-30T06:30:09.753ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
81642025-04-30T06:30:09.753ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81652025-04-30T06:30:09.753ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81662025-04-30T06:30:09.754ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81672025-04-30T06:30:09.754ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
81682025-04-30T06:30:09.755ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81692025-04-30T06:30:09.756ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81702025-04-30T06:30:09.756ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81712025-04-30T06:30:09.756ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
81722025-04-30T06:30:09.757ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
81732025-04-30T06:30:09.758ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e8 91 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58795 req_id = 785ab3c2-be5a-42c4-bfee-16512281531a
81742025-04-30T06:30:09.758ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59537,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58795 req_id = 785ab3c2-be5a-42c4-bfee-16512281531a
81752025-04-30T06:30:09.758ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
81762025-04-30T06:30:09.759ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59537, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58795 req_id = 785ab3c2-be5a-42c4-bfee-16512281531a
81772025-04-30T06:30:09.759ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
81782025-04-30T06:30:09.759ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
81792025-04-30T06:30:09.759ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
81802025-04-30T06:30:09.759ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81812025-04-30T06:30:09.767ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81822025-04-30T06:30:09.767ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81832025-04-30T06:30:09.767ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
81842025-04-30T06:30:09.768ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81852025-04-30T06:30:09.772ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
81862025-04-30T06:30:09.772ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
81872025-04-30T06:30:09.773ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5425fdb8-bf33-4b40-9399-6029c708b698", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
81882025-04-30T06:30:09.773ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
81892025-04-30T06:30:09.773ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81902025-04-30T06:30:09.773ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
81912025-04-30T06:30:09.774ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81922025-04-30T06:30:09.777ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
81932025-04-30T06:30:09.777ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81942025-04-30T06:30:09.777ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e88cfc17-0dab-41ad-8a4c-07017b69ac0b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
81952025-04-30T06:30:09.778ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
81962025-04-30T06:30:09.781ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f7113a0-0a49-4ff7-9873-c9444d2404dc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
81972025-04-30T06:30:09.781ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
81982025-04-30T06:30:09.785ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "554a29f3-636c-4d1e-b3be-46601f14b8b5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
81992025-04-30T06:30:09.785ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
82002025-04-30T06:30:09.788ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a35784a-4bf5-43f2-b427-cb3d8479e9fd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82012025-04-30T06:30:09.788ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
82022025-04-30T06:30:09.792ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc279667-8b25-44ac-a546-ebd3b483f617", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82032025-04-30T06:30:09.792ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
82042025-04-30T06:30:09.796ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f169f65-34dc-48ab-be5a-960c011e0bc2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82052025-04-30T06:30:09.796ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
82062025-04-30T06:30:09.800ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f506a72-f6c0-4226-93c2-86ede5834309", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82072025-04-30T06:30:09.800ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
82082025-04-30T06:30:09.801ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a933ac29-6031-4bfb-8062-73a9d948aef3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82092025-04-30T06:30:09.801ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
82102025-04-30T06:30:09.805ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
82112025-04-30T06:30:09.805ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f2847a9-0f22-416d-8b7b-9700b5188e97", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82122025-04-30T06:30:09.806ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
82132025-04-30T06:30:09.807ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57d9bb08-db51-4bfc-8dd2-c11b481a15b3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82142025-04-30T06:30:09.807ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
82152025-04-30T06:30:09.809ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf3ce79d-e912-4247-894b-064b96664984", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82162025-04-30T06:30:09.809ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
82172025-04-30T06:30:09.810ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9479971a-9fa6-4197-ba1f-12803089b2c5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82182025-04-30T06:30:09.811ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
82192025-04-30T06:30:09.814ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b48fa8b-4c2b-4aa7-bfe5-e4a688e5b8fb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82202025-04-30T06:30:09.814ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
82212025-04-30T06:30:09.816ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a85a61b-5e85-4260-9bf6-6d80c401f01c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82222025-04-30T06:30:09.818ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
82232025-04-30T06:30:09.824ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fc113b5-86e2-460b-95a9-ea344f989d55", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82242025-04-30T06:30:09.825ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
82252025-04-30T06:30:09.829ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f36fcc46-f984-4322-b28b-b1432f8aaa06", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82262025-04-30T06:30:09.829ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
82272025-04-30T06:30:09.835ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2667b3f3-0f08-4d6d-9ec9-15436c2d8a5f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82282025-04-30T06:30:09.836ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
82292025-04-30T06:30:09.846ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bc5a023-6203-437a-90ec-22142b57f60b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82302025-04-30T06:30:09.846ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
82312025-04-30T06:30:09.848ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4883caba-cdad-4e01-89a9-3d98bb64d430", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82322025-04-30T06:30:09.849ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
82332025-04-30T06:30:09.851ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b5b53f7-1082-41c9-8e2e-cb9c9f56fd22", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82342025-04-30T06:30:09.851ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
82352025-04-30T06:30:09.852ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13c32c8a-f88e-46fc-a8f3-47b38bee4354", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82362025-04-30T06:30:09.853ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
82372025-04-30T06:30:09.855ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82382025-04-30T06:30:09.856ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82392025-04-30T06:30:09.856ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82402025-04-30T06:30:09.856ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
82412025-04-30T06:30:09.858ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82422025-04-30T06:30:09.858ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ead878cd-f1bb-45cb-a485-10baa87337ac", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82432025-04-30T06:30:09.858ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
82442025-04-30T06:30:09.860ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
82452025-04-30T06:30:09.861ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85c138a5-a774-49e5-9003-b9fbccdbc3e6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82462025-04-30T06:30:09.861ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
82472025-04-30T06:30:09.862ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82482025-04-30T06:30:09.870ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82492025-04-30T06:30:09.870ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82502025-04-30T06:30:09.870ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
82512025-04-30T06:30:09.871ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82522025-04-30T06:30:09.877ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82532025-04-30T06:30:09.877ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
82542025-04-30T06:30:09.878ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82552025-04-30T06:30:09.879ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b496346-3943-4e7f-b3c7-a05e14b1382d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82562025-04-30T06:30:09.879ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
82572025-04-30T06:30:09.883ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21de4861-9760-4bfc-8dfb-f83eedb77616", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82582025-04-30T06:30:09.884ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
82592025-04-30T06:30:09.892ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93125798-c030-4127-8081-a2edf3610353", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82602025-04-30T06:30:09.893ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
82612025-04-30T06:30:09.899ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0cfb4ba-1941-437e-a24a-258e132d3cac", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82622025-04-30T06:30:09.900ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
82632025-04-30T06:30:09.904ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebd835f3-50ff-456b-a093-b1107f2a19e2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82642025-04-30T06:30:09.905ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
82652025-04-30T06:30:09.913ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06b3c655-1d88-4f3b-a9f9-33b2d268e2c1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82662025-04-30T06:30:09.913ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
82672025-04-30T06:30:09.915ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "165d05ec-6c56-46b4-a125-b1b7abeae94f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82682025-04-30T06:30:09.915ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
82692025-04-30T06:30:09.921ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
82702025-04-30T06:30:09.923ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f83d70a0-c54a-46db-b0e1-0cc45ec8c483", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82712025-04-30T06:30:09.923ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
82722025-04-30T06:30:09.924ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86f05fae-817a-41e9-9741-a125d5dde50a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82732025-04-30T06:30:09.925ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
82742025-04-30T06:30:09.926ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e25ed9c7-2973-4462-ab16-8d4c422685b0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82752025-04-30T06:30:09.927ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
82762025-04-30T06:30:09.928ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd96742c-9d45-462a-bbb7-d5c1513e4bf3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82772025-04-30T06:30:09.928ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
82782025-04-30T06:30:09.931ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77ad4967-f93c-4647-ac06-23521c3ea80b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82792025-04-30T06:30:09.931ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
82802025-04-30T06:30:09.933ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4c9765e-a9a4-4d8e-80b5-ebb821b99860", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82812025-04-30T06:30:09.933ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
82822025-04-30T06:30:09.935ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ead9e62-5651-4747-8b27-9f8e92df098f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82832025-04-30T06:30:09.936ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
82842025-04-30T06:30:09.938ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b49ea694-2a8c-45d7-b34c-fd5ceb8fa3aa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82852025-04-30T06:30:09.939ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
82862025-04-30T06:30:09.941ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9ea7be9-4dea-40f4-88f8-a568c114af34", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82872025-04-30T06:30:09.942ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
82882025-04-30T06:30:09.946ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96be718d-868d-43e1-a3e9-5d81b138ff64", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82892025-04-30T06:30:09.946ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
82902025-04-30T06:30:09.948ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e81912ef-61af-44b1-8c90-091e2794c414", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82912025-04-30T06:30:09.948ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
82922025-04-30T06:30:09.967ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8077060b-984a-4b5d-839d-78ebdd7ef1b0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
82932025-04-30T06:30:09.967ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
82942025-04-30T06:30:09.974ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82952025-04-30T06:30:09.974ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82962025-04-30T06:30:09.974ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82972025-04-30T06:30:09.974ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
82982025-04-30T06:30:09.975ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82992025-04-30T06:30:09.996ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83002025-04-30T06:30:10.009ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83012025-04-30T06:30:10.009ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83022025-04-30T06:30:10.009ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
83032025-04-30T06:30:10.010ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83042025-04-30T06:30:10.011ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e0e5950-e160-4ea3-9c1b-31e235506a40", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
83052025-04-30T06:30:10.012ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
83062025-04-30T06:30:10.021ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83072025-04-30T06:30:10.021ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
83082025-04-30T06:30:10.022ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83092025-04-30T06:30:10.038ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fffb5c2-9f77-4248-bbb8-3ffa383dd52e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:09 GMT"} })
83102025-04-30T06:30:10.038ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83112025-04-30T06:30:10.038ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83122025-04-30T06:30:10.038ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83132025-04-30T06:30:10.038ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83142025-04-30T06:30:10.041ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83152025-04-30T06:30:10.047ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83162025-04-30T06:30:10.053ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83172025-04-30T06:30:10.053ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83182025-04-30T06:30:10.053ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83192025-04-30T06:30:10.053ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83202025-04-30T06:30:10.053ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83212025-04-30T06:30:10.054ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83222025-04-30T06:30:10.082ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83232025-04-30T06:30:10.087ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83242025-04-30T06:30:10.087ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83252025-04-30T06:30:10.087ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83262025-04-30T06:30:10.088ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83272025-04-30T06:30:10.158ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
83282025-04-30T06:30:10.158ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
83292025-04-30T06:30:10.159ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:41407 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
83302025-04-30T06:30:10.159ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:41407 req_id = b828ccdf-8f64-46ec-9670-3d7937b61222 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83312025-04-30T06:30:10.160ZINFOtest_image_promotion (dropshot): request completed latency_us = 554 local_addr = [::1]:60070 method = GET remote_addr = [::1]:41407 req_id = b828ccdf-8f64-46ec-9670-3d7937b61222 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83322025-04-30T06:30:10.161ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b828ccdf-8f64-46ec-9670-3d7937b61222", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
83332025-04-30T06:30:10.163ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83342025-04-30T06:30:10.163ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83352025-04-30T06:30:10.163ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
83362025-04-30T06:30:10.165ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
83372025-04-30T06:30:10.165ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83382025-04-30T06:30:10.165ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83392025-04-30T06:30:10.165ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83402025-04-30T06:30:10.166ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83412025-04-30T06:30:10.171ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83422025-04-30T06:30:10.182ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
83432025-04-30T06:30:10.183ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83442025-04-30T06:30:10.183ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83452025-04-30T06:30:10.184ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83462025-04-30T06:30:10.200ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
83472025-04-30T06:30:10.228ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
83482025-04-30T06:30:10.229ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
83492025-04-30T06:30:10.229ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 570.263192ms iteration = 27 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
83502025-04-30T06:30:10.229ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 28 reason = Signaled
83512025-04-30T06:30:10.229ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
83522025-04-30T06:30:10.229ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83532025-04-30T06:30:10.229ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83542025-04-30T06:30:10.229ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83552025-04-30T06:30:10.229ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83562025-04-30T06:30:10.233ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83572025-04-30T06:30:10.238ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
83582025-04-30T06:30:10.238ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83592025-04-30T06:30:10.238ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83602025-04-30T06:30:10.239ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83612025-04-30T06:30:10.242ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
83622025-04-30T06:30:10.242ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
83632025-04-30T06:30:10.243ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 7c 49 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 |I..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52609 req_id = 302c0f72-b7d8-4b6c-95ba-2972f2ce1807
83642025-04-30T06:30:10.244ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31817,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52609 req_id = 302c0f72-b7d8-4b6c-95ba-2972f2ce1807
83652025-04-30T06:30:10.244ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
83662025-04-30T06:30:10.244ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
83672025-04-30T06:30:10.244ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52609 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 302c0f72-b7d8-4b6c-95ba-2972f2ce1807
83682025-04-30T06:30:10.244ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3475, tv_nsec: 43708000 } })
83692025-04-30T06:30:10.244ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
83702025-04-30T06:30:10.245ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: af 35 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .5..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55439 req_id = 2cfdae22-f46c-4b63-9c65-63927d561677
83712025-04-30T06:30:10.245ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44853,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55439 req_id = 2cfdae22-f46c-4b63-9c65-63927d561677
83722025-04-30T06:30:10.245ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
83732025-04-30T06:30:10.245ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55439 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2cfdae22-f46c-4b63-9c65-63927d561677
83742025-04-30T06:30:10.301ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
83752025-04-30T06:30:10.301ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83762025-04-30T06:30:10.301ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83772025-04-30T06:30:10.301ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83782025-04-30T06:30:10.301ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
83792025-04-30T06:30:10.302ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83802025-04-30T06:30:10.302ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e2 61 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .a..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57366 req_id = 37d5f61e-da2f-4d13-a0a3-392bbacee407
83812025-04-30T06:30:10.302ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57953,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57366 req_id = 37d5f61e-da2f-4d13-a0a3-392bbacee407
83822025-04-30T06:30:10.303ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
83832025-04-30T06:30:10.303ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57953, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57366 req_id = 37d5f61e-da2f-4d13-a0a3-392bbacee407
83842025-04-30T06:30:10.306ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
83852025-04-30T06:30:10.306ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
83862025-04-30T06:30:10.306ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
83872025-04-30T06:30:10.306ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83882025-04-30T06:30:10.311ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83892025-04-30T06:30:10.311ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83902025-04-30T06:30:10.312ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
83912025-04-30T06:30:10.312ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83922025-04-30T06:30:10.314ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83932025-04-30T06:30:10.314ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
83942025-04-30T06:30:10.315ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83952025-04-30T06:30:10.316ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2c1f8b4-07f9-4d35-83f5-b03f9507bd9c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
83962025-04-30T06:30:10.316ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
83972025-04-30T06:30:10.321ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02b7303a-cddc-44a9-ad0f-6151b65f9724", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
83982025-04-30T06:30:10.322ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
83992025-04-30T06:30:10.323ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55df37be-cf16-41f3-8aa2-329ad3da541e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84002025-04-30T06:30:10.324ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
84012025-04-30T06:30:10.326ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0112fa1c-7dc1-441f-a085-5f6bec305349", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84022025-04-30T06:30:10.326ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
84032025-04-30T06:30:10.330ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a44d10f6-2510-449e-a79a-4ae834748712", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84042025-04-30T06:30:10.331ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
84052025-04-30T06:30:10.334ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad48a226-6198-4ba9-8f5d-89c25f8baf93", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84062025-04-30T06:30:10.334ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
84072025-04-30T06:30:10.335ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d07115a0-d8fa-4aa6-8b01-704b208aa1f1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84082025-04-30T06:30:10.335ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
84092025-04-30T06:30:10.338ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fbe5b1f-48f6-4ea8-bb7b-c8c6a38dcb91", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84102025-04-30T06:30:10.338ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
84112025-04-30T06:30:10.343ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee996157-8c6f-4f1d-9525-e3c8c91bcf2a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84122025-04-30T06:30:10.343ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
84132025-04-30T06:30:10.349ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77aed480-106a-4dc0-872f-8d43a75be032", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84142025-04-30T06:30:10.350ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
84152025-04-30T06:30:10.350ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
84162025-04-30T06:30:10.352ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3183d88-16d5-4820-9167-9fe391a7863b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84172025-04-30T06:30:10.352ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
84182025-04-30T06:30:10.354ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "576eec53-dcb6-4d5b-9f5e-1fb7f0c1873d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84192025-04-30T06:30:10.355ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
84202025-04-30T06:30:10.358ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "708a7d05-9731-42db-935a-883c4ee8fe1c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84212025-04-30T06:30:10.358ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
84222025-04-30T06:30:10.362ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18a3b7a7-e343-4727-b241-5cbb18731c37", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84232025-04-30T06:30:10.362ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
84242025-04-30T06:30:10.365ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0623437-4431-4b0a-a8fc-e182ac18752e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84252025-04-30T06:30:10.365ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
84262025-04-30T06:30:10.367ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "78c91cae-bac3-4f57-8208-ed456b78b6e1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84272025-04-30T06:30:10.367ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
84282025-04-30T06:30:10.370ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "794caa5e-3a8a-49a6-9866-8318de3d6289", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84292025-04-30T06:30:10.370ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
84302025-04-30T06:30:10.374ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1fac830-c2b4-4e31-83ab-8e384eb5e59a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84312025-04-30T06:30:10.374ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
84322025-04-30T06:30:10.377ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dbb80ea-da07-43f0-b1b8-ae0e03a2a7e4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84332025-04-30T06:30:10.377ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
84342025-04-30T06:30:10.377ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "546601a2-3988-453a-8d7e-74c2107b0793", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84352025-04-30T06:30:10.378ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
84362025-04-30T06:30:10.379ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "620047b8-97a4-4b1e-be5a-f6a64998ec48", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84372025-04-30T06:30:10.379ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
84382025-04-30T06:30:10.380ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fc1dbc7-2b47-4dff-a19f-c3b95c817669", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84392025-04-30T06:30:10.380ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
84402025-04-30T06:30:10.382ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2716737b-508c-4488-b678-d4bd9ba6307c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84412025-04-30T06:30:10.382ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
84422025-04-30T06:30:10.384ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "011550f5-41f5-4ab5-98c0-40fef22e76ed", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84432025-04-30T06:30:10.384ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
84442025-04-30T06:30:10.387ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12e1406f-2fa6-4a2d-a246-2260d68dcdc4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84452025-04-30T06:30:10.387ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
84462025-04-30T06:30:10.390ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "000ef68a-691c-43f8-97c7-1723a3eb30e7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84472025-04-30T06:30:10.390ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
84482025-04-30T06:30:10.394ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "890aacb9-e5d8-44f0-bd9e-cdc02e86a376", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84492025-04-30T06:30:10.395ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
84502025-04-30T06:30:10.397ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0a58eac-e3b2-420d-b695-d431338d9f18", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84512025-04-30T06:30:10.397ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
84522025-04-30T06:30:10.402ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84532025-04-30T06:30:10.402ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84542025-04-30T06:30:10.402ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84552025-04-30T06:30:10.402ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
84562025-04-30T06:30:10.403ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84572025-04-30T06:30:10.406ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84582025-04-30T06:30:10.415ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84592025-04-30T06:30:10.415ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84602025-04-30T06:30:10.415ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
84612025-04-30T06:30:10.415ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84622025-04-30T06:30:10.418ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19051574-8695-488a-9a11-8f81a8dd885b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84632025-04-30T06:30:10.418ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
84642025-04-30T06:30:10.419ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84652025-04-30T06:30:10.419ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
84662025-04-30T06:30:10.419ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84672025-04-30T06:30:10.422ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4373e43-d8a9-44bc-b0a4-75b1b03c7bd3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84682025-04-30T06:30:10.422ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
84692025-04-30T06:30:10.423ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6871e73-a578-4439-ae1a-5e0d4afa9bbc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84702025-04-30T06:30:10.424ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
84712025-04-30T06:30:10.430ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ece1675b-771f-476c-a2ef-efb31e3f6d5b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84722025-04-30T06:30:10.431ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
84732025-04-30T06:30:10.435ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5aa819e-4d8a-40a1-a828-b8518f7b9714", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84742025-04-30T06:30:10.436ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
84752025-04-30T06:30:10.440ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08d85622-4e17-4b12-950c-e25261a27069", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84762025-04-30T06:30:10.441ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
84772025-04-30T06:30:10.445ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c67f935c-8b2a-490e-81a1-2fab3182a36f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84782025-04-30T06:30:10.445ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
84792025-04-30T06:30:10.447ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb10b310-ae0b-43d1-908e-b0d59dc8d6d7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84802025-04-30T06:30:10.447ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
84812025-04-30T06:30:10.452ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d78763bf-0144-4ba1-a05d-bb85e6864fcc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84822025-04-30T06:30:10.452ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
84832025-04-30T06:30:10.453ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
84842025-04-30T06:30:10.456ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d854838-f7a0-4e2e-8938-422ef1666f00", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84852025-04-30T06:30:10.456ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
84862025-04-30T06:30:10.466ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c29d5a4d-3151-40fc-a0fd-9745f0e80118", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84872025-04-30T06:30:10.466ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
84882025-04-30T06:30:10.468ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "937c8ed3-995f-45c7-b5a6-de390da8f653", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84892025-04-30T06:30:10.468ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
84902025-04-30T06:30:10.478ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7253c03-f401-4e20-82d4-c1326d9871fc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84912025-04-30T06:30:10.479ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
84922025-04-30T06:30:10.486ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17e66662-1d50-49cd-a558-e60479e1e2df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84932025-04-30T06:30:10.486ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
84942025-04-30T06:30:10.491ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4baa5108-356f-4cb5-abaf-448ce50b17ba", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84952025-04-30T06:30:10.492ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
84962025-04-30T06:30:10.493ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f5d15a3-94f2-4bcb-b192-b07ba803864e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84972025-04-30T06:30:10.493ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
84982025-04-30T06:30:10.496ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f3b6802-b9fb-41b6-a2ae-e8d261cfd620", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
84992025-04-30T06:30:10.497ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85002025-04-30T06:30:10.497ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85012025-04-30T06:30:10.497ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85022025-04-30T06:30:10.497ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85032025-04-30T06:30:10.498ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85042025-04-30T06:30:10.503ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85052025-04-30T06:30:10.504ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85062025-04-30T06:30:10.504ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85072025-04-30T06:30:10.504ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
85082025-04-30T06:30:10.504ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85092025-04-30T06:30:10.506ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85102025-04-30T06:30:10.511ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85112025-04-30T06:30:10.511ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85122025-04-30T06:30:10.511ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85132025-04-30T06:30:10.511ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85142025-04-30T06:30:10.511ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85152025-04-30T06:30:10.512ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85162025-04-30T06:30:10.512ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
85172025-04-30T06:30:10.512ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
85182025-04-30T06:30:10.513ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 9f90630a-4271-4f10-a262-180e62bb5806 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
85192025-04-30T06:30:10.514ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 9f90630a-4271-4f10-a262-180e62bb5806 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
85202025-04-30T06:30:10.514ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 9f90630a-4271-4f10-a262-180e62bb5806 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
85212025-04-30T06:30:10.514ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 9f90630a-4271-4f10-a262-180e62bb5806 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
85222025-04-30T06:30:10.515ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85232025-04-30T06:30:10.523ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85242025-04-30T06:30:10.523ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85252025-04-30T06:30:10.523ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
85262025-04-30T06:30:10.526ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85272025-04-30T06:30:10.527ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85282025-04-30T06:30:10.531ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85292025-04-30T06:30:10.531ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85302025-04-30T06:30:10.531ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85312025-04-30T06:30:10.532ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85322025-04-30T06:30:10.534ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 21224 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 9f90630a-4271-4f10-a262-180e62bb5806 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
85332025-04-30T06:30:10.536ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f90630a-4271-4f10-a262-180e62bb5806", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
85342025-04-30T06:30:10.537ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
85352025-04-30T06:30:10.537ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = db668cc8-f3f9-481c-8d69-f9f8a90701c0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
85362025-04-30T06:30:10.538ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = db668cc8-f3f9-481c-8d69-f9f8a90701c0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
85372025-04-30T06:30:10.538ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = db668cc8-f3f9-481c-8d69-f9f8a90701c0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
85382025-04-30T06:30:10.539ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = db668cc8-f3f9-481c-8d69-f9f8a90701c0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
85392025-04-30T06:30:10.613ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
85402025-04-30T06:30:10.613ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
85412025-04-30T06:30:10.614ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85422025-04-30T06:30:10.614ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
85432025-04-30T06:30:10.615ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85442025-04-30T06:30:10.616ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:49341 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
85452025-04-30T06:30:10.616ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85462025-04-30T06:30:10.616ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85472025-04-30T06:30:10.617ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 79801 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = db668cc8-f3f9-481c-8d69-f9f8a90701c0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
85482025-04-30T06:30:10.617ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:49341 req_id = 7f66a7c9-1173-4b56-bee6-53aae687e1fa sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
85492025-04-30T06:30:10.618ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "db668cc8-f3f9-481c-8d69-f9f8a90701c0", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
85502025-04-30T06:30:10.618ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85512025-04-30T06:30:10.618ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
85522025-04-30T06:30:10.618ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
85532025-04-30T06:30:10.619ZINFOtest_image_promotion (dropshot): request completed latency_us = 1623 local_addr = [::1]:60070 method = GET remote_addr = [::1]:49341 req_id = 7f66a7c9-1173-4b56-bee6-53aae687e1fa response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
85542025-04-30T06:30:10.621ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85552025-04-30T06:30:10.621ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85562025-04-30T06:30:10.621ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85572025-04-30T06:30:10.622ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85582025-04-30T06:30:10.622ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f66a7c9-1173-4b56-bee6-53aae687e1fa", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
85592025-04-30T06:30:10.626ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
85602025-04-30T06:30:10.626ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
85612025-04-30T06:30:10.626ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
85622025-04-30T06:30:10.631ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
85632025-04-30T06:30:10.631ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85642025-04-30T06:30:10.631ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85652025-04-30T06:30:10.631ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85662025-04-30T06:30:10.632ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85672025-04-30T06:30:10.637ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85682025-04-30T06:30:10.648ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
85692025-04-30T06:30:10.648ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85702025-04-30T06:30:10.649ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85712025-04-30T06:30:10.651ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85722025-04-30T06:30:10.691ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
85732025-04-30T06:30:10.692ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
85742025-04-30T06:30:10.692ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
85752025-04-30T06:30:10.692ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 463.377289ms iteration = 28 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
85762025-04-30T06:30:10.692ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 29 reason = Signaled
85772025-04-30T06:30:10.692ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
85782025-04-30T06:30:10.692ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85792025-04-30T06:30:10.692ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85802025-04-30T06:30:10.692ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85812025-04-30T06:30:10.693ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85822025-04-30T06:30:10.696ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85832025-04-30T06:30:10.703ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
85842025-04-30T06:30:10.703ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85852025-04-30T06:30:10.703ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85862025-04-30T06:30:10.705ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85872025-04-30T06:30:10.710ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
85882025-04-30T06:30:10.710ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
85892025-04-30T06:30:10.711ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: af 45 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .E..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62392 req_id = 155b594a-147d-4692-9fbb-5265a4d8cc6a
85902025-04-30T06:30:10.711ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44869,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62392 req_id = 155b594a-147d-4692-9fbb-5265a4d8cc6a
85912025-04-30T06:30:10.711ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
85922025-04-30T06:30:10.711ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
85932025-04-30T06:30:10.712ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62392 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 155b594a-147d-4692-9fbb-5265a4d8cc6a
85942025-04-30T06:30:10.712ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3475, tv_nsec: 511329015 } })
85952025-04-30T06:30:10.712ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
85962025-04-30T06:30:10.712ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 32 8c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 2...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59813 req_id = 21d53c08-bd34-4846-b41a-dea7ec599679
85972025-04-30T06:30:10.712ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12940,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59813 req_id = 21d53c08-bd34-4846-b41a-dea7ec599679
85982025-04-30T06:30:10.712ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
85992025-04-30T06:30:10.713ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59813 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 21d53c08-bd34-4846-b41a-dea7ec599679
86002025-04-30T06:30:10.761ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
86012025-04-30T06:30:10.761ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86022025-04-30T06:30:10.762ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86032025-04-30T06:30:10.762ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86042025-04-30T06:30:10.762ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
86052025-04-30T06:30:10.763ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86062025-04-30T06:30:10.763ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86072025-04-30T06:30:10.763ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86082025-04-30T06:30:10.763ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
86092025-04-30T06:30:10.768ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
86102025-04-30T06:30:10.769ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ce 60 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .`..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50704 req_id = 9167bda9-003d-440a-91ad-117f97e0f295
86112025-04-30T06:30:10.769ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52832,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50704 req_id = 9167bda9-003d-440a-91ad-117f97e0f295
86122025-04-30T06:30:10.769ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
86132025-04-30T06:30:10.769ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52832, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50704 req_id = 9167bda9-003d-440a-91ad-117f97e0f295
86142025-04-30T06:30:10.769ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
86152025-04-30T06:30:10.770ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
86162025-04-30T06:30:10.770ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
86172025-04-30T06:30:10.770ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86182025-04-30T06:30:10.779ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86192025-04-30T06:30:10.779ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86202025-04-30T06:30:10.779ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
86212025-04-30T06:30:10.779ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86222025-04-30T06:30:10.785ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
86232025-04-30T06:30:10.785ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
86242025-04-30T06:30:10.785ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86252025-04-30T06:30:10.785ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
86262025-04-30T06:30:10.787ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86272025-04-30T06:30:10.790ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8e78577-d116-4222-bf5f-52ffce8e3178", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86282025-04-30T06:30:10.791ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
86292025-04-30T06:30:10.793ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
86302025-04-30T06:30:10.793ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86312025-04-30T06:30:10.794ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be81e2c2-3e65-403c-9263-366d8e47ab3d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86322025-04-30T06:30:10.795ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
86332025-04-30T06:30:10.797ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13e2698b-aadf-443d-8fc5-3a6cf550e581", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86342025-04-30T06:30:10.797ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
86352025-04-30T06:30:10.799ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9f9204d-443f-4236-a2d0-b54548e51f3d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86362025-04-30T06:30:10.799ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
86372025-04-30T06:30:10.802ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c99a15d4-ec2a-4251-a1a7-52874153bb78", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86382025-04-30T06:30:10.802ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
86392025-04-30T06:30:10.805ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ab428ed-89f0-4171-a618-cabda7d1e3bf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86402025-04-30T06:30:10.805ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
86412025-04-30T06:30:10.808ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6229c57e-72e2-4b41-89af-a7d38bf2f666", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86422025-04-30T06:30:10.808ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
86432025-04-30T06:30:10.810ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07125aca-fe5f-47f5-9311-ae93cbe31f40", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86442025-04-30T06:30:10.810ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
86452025-04-30T06:30:10.811ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db5428d3-9a67-4627-9910-6f48159f8a0c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86462025-04-30T06:30:10.812ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
86472025-04-30T06:30:10.813ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4d31a02-2e54-4906-9e2b-a84f60c5742f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86482025-04-30T06:30:10.815ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
86492025-04-30T06:30:10.818ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c1ddd20-4600-49b3-939b-e685955c07ca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86502025-04-30T06:30:10.818ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
86512025-04-30T06:30:10.819ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c4a88fc-7c53-46dd-940b-1e1097cb7cbd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86522025-04-30T06:30:10.820ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
86532025-04-30T06:30:10.821ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "770b2cd3-191f-40cf-abcf-895cfefc5c83", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86542025-04-30T06:30:10.821ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
86552025-04-30T06:30:10.826ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9155bd4-1c2d-48de-9f0b-7bf1715b9e76", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86562025-04-30T06:30:10.826ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
86572025-04-30T06:30:10.830ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
86582025-04-30T06:30:10.833ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2ca29d9-8ad1-46f8-aa64-3900f12285bb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86592025-04-30T06:30:10.833ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
86602025-04-30T06:30:10.836ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b98901ef-57aa-483a-9659-bf0bfe4c097e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86612025-04-30T06:30:10.836ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
86622025-04-30T06:30:10.842ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e850cf3-e20c-41cd-8b47-4f2cf6203419", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86632025-04-30T06:30:10.842ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
86642025-04-30T06:30:10.842ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "373d7127-f134-47fe-87bf-7885c5ff4219", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86652025-04-30T06:30:10.842ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
86662025-04-30T06:30:10.845ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59d704d8-ed93-4f09-91b0-f84ef702c46c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86672025-04-30T06:30:10.845ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
86682025-04-30T06:30:10.846ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f542f6a6-feb4-4832-8bae-4b1e06c9f9b8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86692025-04-30T06:30:10.846ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
86702025-04-30T06:30:10.847ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4039b6b9-5a0d-4e5e-9ba1-b5472507b045", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86712025-04-30T06:30:10.847ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
86722025-04-30T06:30:10.848ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a50004e-097c-4916-8cc1-e47c6df5c33b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86732025-04-30T06:30:10.848ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
86742025-04-30T06:30:10.849ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3aedd2ef-b1a9-4c16-b0cd-bf1887bb34ff", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86752025-04-30T06:30:10.849ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
86762025-04-30T06:30:10.851ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "325bfa69-8cf8-4942-b463-579a67deed24", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86772025-04-30T06:30:10.851ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
86782025-04-30T06:30:10.853ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6a006b4-e9f4-4a7c-ab83-49d48f0c537c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86792025-04-30T06:30:10.853ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
86802025-04-30T06:30:10.856ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "184e4cc2-f599-4be8-8408-5b98c7a4afcc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86812025-04-30T06:30:10.856ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
86822025-04-30T06:30:10.858ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64a999ec-4d78-45e8-aef9-048d5bafedc2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86832025-04-30T06:30:10.859ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
86842025-04-30T06:30:10.861ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d6fbebd-060e-438e-8253-75dc53cd1c05", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86852025-04-30T06:30:10.861ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
86862025-04-30T06:30:10.863ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4e1dfbc-1293-4bb8-8242-750cd196a130", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86872025-04-30T06:30:10.863ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
86882025-04-30T06:30:10.864ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
86892025-04-30T06:30:10.865ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "820fa98d-78fc-46a2-a51e-31dfe352a075", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86902025-04-30T06:30:10.866ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
86912025-04-30T06:30:10.867ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3c4bd42-2741-426f-9542-d1992f004198", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86922025-04-30T06:30:10.867ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
86932025-04-30T06:30:10.868ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24658c52-82fe-4add-86dc-6a65ee396071", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86942025-04-30T06:30:10.868ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
86952025-04-30T06:30:10.870ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4b49a99-b9cd-46e2-bc10-a4b5a62140bf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86962025-04-30T06:30:10.870ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
86972025-04-30T06:30:10.871ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35f9a213-2f51-4ac4-b1c6-87c2599c177d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
86982025-04-30T06:30:10.871ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
86992025-04-30T06:30:10.873ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0b08a9c-da67-43b1-a973-fc3fcff46ff3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
87002025-04-30T06:30:10.873ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
87012025-04-30T06:30:10.876ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "134c6a93-cd66-4a08-943b-a1171b6f29d7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
87022025-04-30T06:30:10.876ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
87032025-04-30T06:30:10.879ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97a0f096-77a8-4926-83a3-6fdf61797432", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
87042025-04-30T06:30:10.880ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
87052025-04-30T06:30:10.881ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87062025-04-30T06:30:10.881ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87072025-04-30T06:30:10.881ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87082025-04-30T06:30:10.881ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
87092025-04-30T06:30:10.881ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87102025-04-30T06:30:10.885ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87112025-04-30T06:30:10.891ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87122025-04-30T06:30:10.891ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87132025-04-30T06:30:10.891ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
87142025-04-30T06:30:10.892ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87152025-04-30T06:30:10.892ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48822fd1-86a8-472c-868a-0ea63fa92713", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
87162025-04-30T06:30:10.892ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
87172025-04-30T06:30:10.896ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87182025-04-30T06:30:10.896ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
87192025-04-30T06:30:10.896ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87202025-04-30T06:30:10.898ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8014629a-8294-4262-abd7-1612f80dcbea", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
87212025-04-30T06:30:10.898ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
87222025-04-30T06:30:10.901ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b05028c7-bb03-4a5c-bc58-9d9215a091b8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
87232025-04-30T06:30:10.902ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
87242025-04-30T06:30:10.909ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dab903b-3e0e-4e30-9e7c-b5a82bb43c88", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
87252025-04-30T06:30:10.909ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
87262025-04-30T06:30:10.911ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "174f7fe2-eba6-4a7c-9851-a62d42c21155", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
87272025-04-30T06:30:10.911ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
87282025-04-30T06:30:10.914ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc675ed8-8a39-46a5-b44b-810220ff7029", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
87292025-04-30T06:30:10.914ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
87302025-04-30T06:30:10.916ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b2ce87f-cbc0-4aa1-8734-a3b05291d4d3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
87312025-04-30T06:30:10.916ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
87322025-04-30T06:30:10.918ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e09a07fa-063d-499b-98b5-681f3f440cf3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
87332025-04-30T06:30:10.918ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
87342025-04-30T06:30:10.918ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87352025-04-30T06:30:10.919ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87362025-04-30T06:30:10.919ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87372025-04-30T06:30:10.920ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87382025-04-30T06:30:10.929ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87392025-04-30T06:30:10.941ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
87402025-04-30T06:30:10.942ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
87412025-04-30T06:30:10.942ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87422025-04-30T06:30:10.943ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87432025-04-30T06:30:10.943ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87442025-04-30T06:30:10.944ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87452025-04-30T06:30:10.967ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
87462025-04-30T06:30:10.967ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87472025-04-30T06:30:10.976ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
87482025-04-30T06:30:10.977ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87492025-04-30T06:30:10.977ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87502025-04-30T06:30:10.978ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87512025-04-30T06:30:11.060ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
87522025-04-30T06:30:11.060ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
87532025-04-30T06:30:11.060ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87542025-04-30T06:30:11.061ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87552025-04-30T06:30:11.061ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87562025-04-30T06:30:11.061ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
87572025-04-30T06:30:11.062ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87582025-04-30T06:30:11.063ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:38093 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
87592025-04-30T06:30:11.070ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:38093 req_id = f5c1eea7-f2e7-4e2d-aa5a-a18765eea23a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
87602025-04-30T06:30:11.071ZINFOtest_image_promotion (dropshot): request completed latency_us = 1082 local_addr = [::1]:60070 method = GET remote_addr = [::1]:38093 req_id = f5c1eea7-f2e7-4e2d-aa5a-a18765eea23a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
87612025-04-30T06:30:11.075ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5c1eea7-f2e7-4e2d-aa5a-a18765eea23a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:10 GMT"} })
87622025-04-30T06:30:11.080ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
87632025-04-30T06:30:11.080ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
87642025-04-30T06:30:11.080ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
87652025-04-30T06:30:11.086ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
87662025-04-30T06:30:11.086ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87672025-04-30T06:30:11.086ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87682025-04-30T06:30:11.086ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87692025-04-30T06:30:11.087ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87702025-04-30T06:30:11.088ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87712025-04-30T06:30:11.096ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87722025-04-30T06:30:11.096ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87732025-04-30T06:30:11.096ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
87742025-04-30T06:30:11.097ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87752025-04-30T06:30:11.101ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87762025-04-30T06:30:11.105ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
87772025-04-30T06:30:11.105ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87782025-04-30T06:30:11.105ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87792025-04-30T06:30:11.106ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87802025-04-30T06:30:11.113ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87812025-04-30T06:30:11.113ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
87822025-04-30T06:30:11.115ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87832025-04-30T06:30:11.185ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
87842025-04-30T06:30:11.190ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
87852025-04-30T06:30:11.190ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 497.45689ms iteration = 29 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
87862025-04-30T06:30:11.190ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 30 reason = Signaled
87872025-04-30T06:30:11.190ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
87882025-04-30T06:30:11.190ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87892025-04-30T06:30:11.190ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87902025-04-30T06:30:11.190ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87912025-04-30T06:30:11.194ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87922025-04-30T06:30:11.204ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87932025-04-30T06:30:11.214ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
87942025-04-30T06:30:11.214ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87952025-04-30T06:30:11.214ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87962025-04-30T06:30:11.215ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87972025-04-30T06:30:11.220ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
87982025-04-30T06:30:11.220ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
87992025-04-30T06:30:11.221ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
88002025-04-30T06:30:11.221ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 2f 89 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 /............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62042 req_id = aa041be8-6df2-411d-a8b3-799acc912bf1
88012025-04-30T06:30:11.221ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12169,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62042 req_id = aa041be8-6df2-411d-a8b3-799acc912bf1
88022025-04-30T06:30:11.222ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
88032025-04-30T06:30:11.222ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88042025-04-30T06:30:11.222ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62042 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = aa041be8-6df2-411d-a8b3-799acc912bf1
88052025-04-30T06:30:11.223ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3476, tv_nsec: 22280202 } })
88062025-04-30T06:30:11.223ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
88072025-04-30T06:30:11.223ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 02 4a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .J..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64232 req_id = c28706d2-137f-48e2-980d-3420395fc94d
88082025-04-30T06:30:11.223ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 586,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64232 req_id = c28706d2-137f-48e2-980d-3420395fc94d
88092025-04-30T06:30:11.225ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88102025-04-30T06:30:11.225ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64232 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c28706d2-137f-48e2-980d-3420395fc94d
88112025-04-30T06:30:11.278ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
88122025-04-30T06:30:11.279ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88132025-04-30T06:30:11.279ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88142025-04-30T06:30:11.280ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88152025-04-30T06:30:11.280ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
88162025-04-30T06:30:11.281ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88172025-04-30T06:30:11.282ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5a 63 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c Zc..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50580 req_id = 2aa3a33d-1efe-4c3c-bd2d-81182edfecd6
88182025-04-30T06:30:11.282ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23139,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50580 req_id = 2aa3a33d-1efe-4c3c-bd2d-81182edfecd6
88192025-04-30T06:30:11.282ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
88202025-04-30T06:30:11.282ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23139, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50580 req_id = 2aa3a33d-1efe-4c3c-bd2d-81182edfecd6
88212025-04-30T06:30:11.283ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
88222025-04-30T06:30:11.283ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
88232025-04-30T06:30:11.283ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
88242025-04-30T06:30:11.284ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88252025-04-30T06:30:11.292ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
88262025-04-30T06:30:11.292ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88272025-04-30T06:30:11.292ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
88282025-04-30T06:30:11.293ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88292025-04-30T06:30:11.296ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8582d84-d3c3-4ae9-994c-d416236ad86c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88302025-04-30T06:30:11.296ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
88312025-04-30T06:30:11.296ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88322025-04-30T06:30:11.296ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
88332025-04-30T06:30:11.297ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88342025-04-30T06:30:11.299ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1bda59c-f057-4ae1-9d47-f634e5b21e54", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88352025-04-30T06:30:11.299ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
88362025-04-30T06:30:11.302ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "712e109e-48c9-487b-b4c7-ec9a296e1361", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88372025-04-30T06:30:11.302ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
88382025-04-30T06:30:11.304ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bc7e9aa-4836-413a-ad33-1a0a5ede7eb6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88392025-04-30T06:30:11.304ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
88402025-04-30T06:30:11.307ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c538da8d-604c-454e-9abb-90aa6e214553", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88412025-04-30T06:30:11.307ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
88422025-04-30T06:30:11.308ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cff4208-b704-4f8b-86bd-7ba022160871", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88432025-04-30T06:30:11.309ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
88442025-04-30T06:30:11.312ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b19e0d8-952f-45ac-97d5-a7b047c3eb81", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88452025-04-30T06:30:11.313ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
88462025-04-30T06:30:11.319ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5cd20d7-6e7a-4191-9b2a-acc352653205", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88472025-04-30T06:30:11.319ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
88482025-04-30T06:30:11.321ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c5d3947-26a9-4a7f-90c3-7fe84111b2c5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88492025-04-30T06:30:11.321ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
88502025-04-30T06:30:11.329ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "696c4e24-0724-4ec8-b2f5-a69e9cfadd6b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88512025-04-30T06:30:11.329ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
88522025-04-30T06:30:11.332ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b8db8b4-24a0-44a8-9aa6-45a54b4a9bef", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88532025-04-30T06:30:11.333ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
88542025-04-30T06:30:11.338ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
88552025-04-30T06:30:11.339ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e6aa30a-9257-43d9-a314-ccfdaa729d95", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88562025-04-30T06:30:11.342ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
88572025-04-30T06:30:11.345ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a25b6046-7027-4ffc-a44d-64af0603c7e5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88582025-04-30T06:30:11.345ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
88592025-04-30T06:30:11.359ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6d5b36d-942f-4eb9-a0a1-3a9338d13b84", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88602025-04-30T06:30:11.359ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
88612025-04-30T06:30:11.362ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1711babb-f4d2-4dde-a1e2-daff77684b66", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88622025-04-30T06:30:11.362ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
88632025-04-30T06:30:11.365ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8942f3fc-35e7-4e8e-aacd-f69b3454976e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88642025-04-30T06:30:11.372ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
88652025-04-30T06:30:11.379ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24448eba-c0f0-4cee-b4bf-7159241a0381", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88662025-04-30T06:30:11.379ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
88672025-04-30T06:30:11.381ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d90ccda-e7a7-4023-86cf-54aad7523c96", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88682025-04-30T06:30:11.381ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
88692025-04-30T06:30:11.382ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55ed1083-3530-4eba-aa21-c88ab6943e74", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88702025-04-30T06:30:11.384ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
88712025-04-30T06:30:11.387ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a936ebd-5108-42d7-91b8-fe833e07f07c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88722025-04-30T06:30:11.387ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
88732025-04-30T06:30:11.389ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "221ba895-a9ac-4c33-b994-d974751904f8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88742025-04-30T06:30:11.389ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
88752025-04-30T06:30:11.390ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88762025-04-30T06:30:11.390ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88772025-04-30T06:30:11.390ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88782025-04-30T06:30:11.390ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
88792025-04-30T06:30:11.391ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88802025-04-30T06:30:11.393ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27ace99a-07f6-4ae7-9cf6-82a9491f52d7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88812025-04-30T06:30:11.394ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
88822025-04-30T06:30:11.395ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88832025-04-30T06:30:11.410ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
88842025-04-30T06:30:11.410ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88852025-04-30T06:30:11.410ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
88862025-04-30T06:30:11.411ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88872025-04-30T06:30:11.414ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "304cdf43-9277-423a-9532-48d2ab41bc63", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88882025-04-30T06:30:11.415ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
88892025-04-30T06:30:11.415ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88902025-04-30T06:30:11.415ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
88912025-04-30T06:30:11.415ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88922025-04-30T06:30:11.417ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12c3b302-4be8-4f0c-806e-3ea26af7f727", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88932025-04-30T06:30:11.417ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
88942025-04-30T06:30:11.422ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62a9f192-1b4d-4684-8a0a-6af5cec807af", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88952025-04-30T06:30:11.422ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
88962025-04-30T06:30:11.432ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09b2420c-10e0-4016-b36d-4bd29f0a67e2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88972025-04-30T06:30:11.433ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
88982025-04-30T06:30:11.439ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e70639d4-d0f8-4753-9ce1-cde727dc96f6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
88992025-04-30T06:30:11.439ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
89002025-04-30T06:30:11.441ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "304981e0-ecf4-4813-9db8-8edf0d91e71d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89012025-04-30T06:30:11.441ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
89022025-04-30T06:30:11.462ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ce5db13-fddb-4336-8156-30881d83dfe4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89032025-04-30T06:30:11.462ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
89042025-04-30T06:30:11.464ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ec83116-c64a-416c-beb8-26edf637951c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89052025-04-30T06:30:11.464ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
89062025-04-30T06:30:11.465ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1646db1e-78e8-490c-839a-6947354b60b1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89072025-04-30T06:30:11.466ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
89082025-04-30T06:30:11.469ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc482a50-fac3-4809-be8b-10f5d8370394", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89092025-04-30T06:30:11.469ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
89102025-04-30T06:30:11.470ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2503ffe-89ce-4b9b-a24e-a26ef52a386c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89112025-04-30T06:30:11.470ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
89122025-04-30T06:30:11.479ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83e96b51-bf09-4612-96c0-ee8b369defda", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89132025-04-30T06:30:11.479ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
89142025-04-30T06:30:11.481ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
89152025-04-30T06:30:11.483ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "edc87f36-3bfa-4ec3-8078-03cfb0c38c0f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89162025-04-30T06:30:11.484ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
89172025-04-30T06:30:11.488ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "144f0b99-cdd2-4756-bdbc-18589e4b0efa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89182025-04-30T06:30:11.489ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
89192025-04-30T06:30:11.494ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27f332bc-0c22-4ce3-98fb-de87982c11ca", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89202025-04-30T06:30:11.494ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
89212025-04-30T06:30:11.497ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0580fca7-eeb3-4d36-a1d8-0c5ee35fee99", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89222025-04-30T06:30:11.497ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
89232025-04-30T06:30:11.504ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c90650e-68d6-4472-a3d9-e1ffa6059624", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89242025-04-30T06:30:11.508ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
89252025-04-30T06:30:11.514ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db5c2426-0b18-4dca-a0c7-861f53907904", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89262025-04-30T06:30:11.514ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
89272025-04-30T06:30:11.517ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5262aedf-92be-4d3b-ab6a-c77db73a52bd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89282025-04-30T06:30:11.517ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
89292025-04-30T06:30:11.520ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e58fc3b-efb3-4254-beb2-3db6e2c2631d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89302025-04-30T06:30:11.520ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
89312025-04-30T06:30:11.523ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57bfab33-6105-4aef-b68b-f1058009d3e2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89322025-04-30T06:30:11.523ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
89332025-04-30T06:30:11.524ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "380f8cb0-219d-4cd5-8121-6114ce748e27", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89342025-04-30T06:30:11.525ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
89352025-04-30T06:30:11.526ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ecd6bef-6c35-474a-809d-a0731094e2a5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89362025-04-30T06:30:11.526ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
89372025-04-30T06:30:11.526ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89382025-04-30T06:30:11.526ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89392025-04-30T06:30:11.526ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89402025-04-30T06:30:11.528ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89412025-04-30T06:30:11.532ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89422025-04-30T06:30:11.532ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89432025-04-30T06:30:11.532ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89442025-04-30T06:30:11.532ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
89452025-04-30T06:30:11.534ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89462025-04-30T06:30:11.537ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89472025-04-30T06:30:11.543ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
89482025-04-30T06:30:11.544ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
89492025-04-30T06:30:11.544ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89502025-04-30T06:30:11.545ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89512025-04-30T06:30:11.545ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89522025-04-30T06:30:11.545ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89532025-04-30T06:30:11.547ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89542025-04-30T06:30:11.547ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89552025-04-30T06:30:11.548ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89562025-04-30T06:30:11.557ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89572025-04-30T06:30:11.557ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89582025-04-30T06:30:11.557ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
89592025-04-30T06:30:11.558ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89602025-04-30T06:30:11.563ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89612025-04-30T06:30:11.571ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
89622025-04-30T06:30:11.571ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89632025-04-30T06:30:11.571ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89642025-04-30T06:30:11.573ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89652025-04-30T06:30:11.574ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89662025-04-30T06:30:11.574ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89672025-04-30T06:30:11.574ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89682025-04-30T06:30:11.574ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89692025-04-30T06:30:11.585ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89702025-04-30T06:30:11.585ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
89712025-04-30T06:30:11.589ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89722025-04-30T06:30:11.708ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
89732025-04-30T06:30:11.708ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
89742025-04-30T06:30:11.710ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:47621 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
89752025-04-30T06:30:11.711ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:47621 req_id = b472b756-d1b0-4781-aeab-c8f5d5dfbdea sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
89762025-04-30T06:30:11.712ZINFOtest_image_promotion (dropshot): request completed latency_us = 1789 local_addr = [::1]:60070 method = GET remote_addr = [::1]:47621 req_id = b472b756-d1b0-4781-aeab-c8f5d5dfbdea response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
89772025-04-30T06:30:11.714ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b472b756-d1b0-4781-aeab-c8f5d5dfbdea", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
89782025-04-30T06:30:11.718ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
89792025-04-30T06:30:11.718ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
89802025-04-30T06:30:11.718ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
89812025-04-30T06:30:11.722ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
89822025-04-30T06:30:11.722ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89832025-04-30T06:30:11.722ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89842025-04-30T06:30:11.722ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89852025-04-30T06:30:11.723ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89862025-04-30T06:30:11.727ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89872025-04-30T06:30:11.732ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
89882025-04-30T06:30:11.733ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89892025-04-30T06:30:11.733ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89902025-04-30T06:30:11.733ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89912025-04-30T06:30:11.751ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
89922025-04-30T06:30:11.752ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
89932025-04-30T06:30:11.753ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
89942025-04-30T06:30:11.753ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
89952025-04-30T06:30:11.760ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
89962025-04-30T06:30:11.761ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
89972025-04-30T06:30:11.762ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
89982025-04-30T06:30:11.764ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
89992025-04-30T06:30:11.764ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
90002025-04-30T06:30:11.768ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
90012025-04-30T06:30:11.768ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
90022025-04-30T06:30:11.768ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 578.435656ms iteration = 30 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
90032025-04-30T06:30:11.768ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 31 reason = Signaled
90042025-04-30T06:30:11.768ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90052025-04-30T06:30:11.768ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90062025-04-30T06:30:11.768ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90072025-04-30T06:30:11.768ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90082025-04-30T06:30:11.769ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90092025-04-30T06:30:11.775ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90102025-04-30T06:30:11.783ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90112025-04-30T06:30:11.783ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90122025-04-30T06:30:11.783ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90132025-04-30T06:30:11.784ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90142025-04-30T06:30:11.788ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
90152025-04-30T06:30:11.788ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
90162025-04-30T06:30:11.789ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 99 d3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51125 req_id = 8add1f4e-c949-4c77-86cc-52cd97efab93
90172025-04-30T06:30:11.789ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39379,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51125 req_id = 8add1f4e-c949-4c77-86cc-52cd97efab93
90182025-04-30T06:30:11.789ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
90192025-04-30T06:30:11.790ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
90202025-04-30T06:30:11.790ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51125 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8add1f4e-c949-4c77-86cc-52cd97efab93
90212025-04-30T06:30:11.791ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3476, tv_nsec: 590679670 } })
90222025-04-30T06:30:11.791ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
90232025-04-30T06:30:11.792ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 03 39 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .9..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58110 req_id = d080561e-15ae-4e1f-8da5-c9b3dadb977d
90242025-04-30T06:30:11.792ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 825,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58110 req_id = d080561e-15ae-4e1f-8da5-c9b3dadb977d
90252025-04-30T06:30:11.792ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
90262025-04-30T06:30:11.792ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58110 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d080561e-15ae-4e1f-8da5-c9b3dadb977d
90272025-04-30T06:30:11.853ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
90282025-04-30T06:30:11.854ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90292025-04-30T06:30:11.854ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90302025-04-30T06:30:11.854ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90312025-04-30T06:30:11.854ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
90322025-04-30T06:30:11.856ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90332025-04-30T06:30:11.856ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
90342025-04-30T06:30:11.857ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f7 6f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .o..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59950 req_id = 1500545e-7326-4bd6-abbf-e0b0b356d3ab
90352025-04-30T06:30:11.857ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63343,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59950 req_id = 1500545e-7326-4bd6-abbf-e0b0b356d3ab
90362025-04-30T06:30:11.857ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
90372025-04-30T06:30:11.857ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63343, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59950 req_id = 1500545e-7326-4bd6-abbf-e0b0b356d3ab
90382025-04-30T06:30:11.857ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
90392025-04-30T06:30:11.857ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
90402025-04-30T06:30:11.857ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
90412025-04-30T06:30:11.862ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90422025-04-30T06:30:11.872ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90432025-04-30T06:30:11.872ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90442025-04-30T06:30:11.872ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
90452025-04-30T06:30:11.873ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90462025-04-30T06:30:11.873ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9b6e001-a5b6-4ff3-a93c-71933d389a63", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90472025-04-30T06:30:11.873ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
90482025-04-30T06:30:11.876ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90492025-04-30T06:30:11.876ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
90502025-04-30T06:30:11.876ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90512025-04-30T06:30:11.877ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e76c0333-b1d9-40f0-bd69-3d75c7a4aa1a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90522025-04-30T06:30:11.877ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
90532025-04-30T06:30:11.884ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "118ef519-69cc-452e-8cd9-00e978c9291b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90542025-04-30T06:30:11.884ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
90552025-04-30T06:30:11.886ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "318cf1da-7312-477b-8e3a-60f032e8d552", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90562025-04-30T06:30:11.886ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
90572025-04-30T06:30:11.890ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f643bdc5-1cfd-47c8-8585-cd5f3591bc59", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90582025-04-30T06:30:11.890ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
90592025-04-30T06:30:11.894ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcfd3892-28ab-4ca4-bc57-2c0a21404fb5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90602025-04-30T06:30:11.894ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
90612025-04-30T06:30:11.897ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1ebdf64-c2f8-4b4d-9e81-4aa27377dba6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90622025-04-30T06:30:11.898ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
90632025-04-30T06:30:11.904ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6dc46d3-0b8f-46cf-b2b2-5288dbfe3999", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90642025-04-30T06:30:11.904ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
90652025-04-30T06:30:11.910ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
90662025-04-30T06:30:11.911ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7c6a1f6-49e0-4938-9c41-beaaa1b14669", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90672025-04-30T06:30:11.911ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
90682025-04-30T06:30:11.913ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8f7f5b3-32f5-412b-83d4-d01b5ae5328d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90692025-04-30T06:30:11.913ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
90702025-04-30T06:30:11.914ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7706ebf1-9ea0-454e-9ae0-5b59c00d8d0c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90712025-04-30T06:30:11.914ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
90722025-04-30T06:30:11.915ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c816f10-1758-4f4c-a52f-285bc1bb11a4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90732025-04-30T06:30:11.915ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
90742025-04-30T06:30:11.917ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "573c3d3c-9dba-4356-b5e1-fdeec2a97bce", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90752025-04-30T06:30:11.917ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
90762025-04-30T06:30:11.919ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7efa62b4-a76f-489f-b035-b67427d420bb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90772025-04-30T06:30:11.919ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
90782025-04-30T06:30:11.922ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2887a0cf-6302-4a65-9694-d2ec1c6dab1c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90792025-04-30T06:30:11.923ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
90802025-04-30T06:30:11.925ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a251482-ca40-4186-becf-d399bf722cf3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90812025-04-30T06:30:11.926ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
90822025-04-30T06:30:11.932ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b252e17-e659-44db-a6dc-d28e039cc239", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90832025-04-30T06:30:11.935ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
90842025-04-30T06:30:11.938ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cbe5054-d0f9-4cd7-94ea-bcde0735b6d1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90852025-04-30T06:30:11.938ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
90862025-04-30T06:30:11.940ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf76d29d-5efd-4218-beb5-975e2f6bc856", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90872025-04-30T06:30:11.940ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
90882025-04-30T06:30:11.941ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b29dc7e1-301e-4dda-aeb4-dd098e30a7c9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90892025-04-30T06:30:11.942ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
90902025-04-30T06:30:11.942ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "635ec9ea-71af-4f5e-96bc-b3a1dc4bb03a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90912025-04-30T06:30:11.943ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
90922025-04-30T06:30:11.944ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c21a059e-3ef7-4a13-bfe7-d8806a854608", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90932025-04-30T06:30:11.944ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
90942025-04-30T06:30:11.945ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "505242f6-a711-484e-b45a-083670e29ef8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90952025-04-30T06:30:11.945ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
90962025-04-30T06:30:11.947ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc47d210-7416-4c80-9846-534f808b2ace", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90972025-04-30T06:30:11.947ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
90982025-04-30T06:30:11.957ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbdea955-518c-482a-8fda-1b4c6237358d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
90992025-04-30T06:30:11.958ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
91002025-04-30T06:30:11.962ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91012025-04-30T06:30:11.962ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91022025-04-30T06:30:11.962ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91032025-04-30T06:30:11.962ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
91042025-04-30T06:30:11.963ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91052025-04-30T06:30:11.979ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91062025-04-30T06:30:11.986ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91072025-04-30T06:30:11.986ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91082025-04-30T06:30:11.986ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
91092025-04-30T06:30:11.986ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91102025-04-30T06:30:11.990ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91112025-04-30T06:30:11.990ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
91122025-04-30T06:30:11.990ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91132025-04-30T06:30:12.012ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53cf0c70-b7b1-4f0b-899d-ece58de5ee1d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91142025-04-30T06:30:12.012ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
91152025-04-30T06:30:12.018ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "721cb078-c3c6-41f9-8873-9ce28d24f02d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91162025-04-30T06:30:12.018ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
91172025-04-30T06:30:12.021ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea43dfe6-f16c-4a95-b0f1-d3630e8c7e12", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91182025-04-30T06:30:12.021ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
91192025-04-30T06:30:12.027ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
91202025-04-30T06:30:12.028ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ded5536-f3d3-4160-9a9b-bcffaea9ef25", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91212025-04-30T06:30:12.028ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
91222025-04-30T06:30:12.030ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32ccd8ef-2c5f-44a8-8f76-50df0341709a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91232025-04-30T06:30:12.030ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
91242025-04-30T06:30:12.031ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9c21d58-f961-4fd0-b07d-fa02fc9f0e2f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91252025-04-30T06:30:12.031ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
91262025-04-30T06:30:12.033ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b960e3e5-7c05-4388-87da-079dcbe00158", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91272025-04-30T06:30:12.033ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
91282025-04-30T06:30:12.035ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cba4d42d-a169-4cc1-af4c-0638b2afdc11", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91292025-04-30T06:30:12.036ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
91302025-04-30T06:30:12.037ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72c31e37-02cc-4269-b6d4-238525e13706", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91312025-04-30T06:30:12.037ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
91322025-04-30T06:30:12.039ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e07aabc-9ef2-4000-bf83-d01c2aa27457", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91332025-04-30T06:30:12.039ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
91342025-04-30T06:30:12.043ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05238fc0-b79c-42fa-a0f1-dace4bf74ace", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91352025-04-30T06:30:12.043ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
91362025-04-30T06:30:12.045ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e562e342-441d-45b3-bfdf-4cda4ffb6b89", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91372025-04-30T06:30:12.045ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
91382025-04-30T06:30:12.047ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "612c7444-0b9a-40ff-85c9-a1e4f41c837f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91392025-04-30T06:30:12.047ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
91402025-04-30T06:30:12.050ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9ad465b-188e-4614-a5ae-68a8ca97d752", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91412025-04-30T06:30:12.050ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
91422025-04-30T06:30:12.052ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ed827cd-a8e6-4d6d-bddb-2157d23ea4b0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91432025-04-30T06:30:12.052ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
91442025-04-30T06:30:12.054ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1babaac8-943e-40c9-82f5-7523816fe88a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91452025-04-30T06:30:12.054ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
91462025-04-30T06:30:12.055ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf10cce0-686a-438a-9615-8a6c6b0a85fa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91472025-04-30T06:30:12.055ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
91482025-04-30T06:30:12.057ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ef3111f-ba60-46a4-831b-a327dce92ba6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91492025-04-30T06:30:12.057ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
91502025-04-30T06:30:12.059ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d21fc979-f4b8-4bb3-adbe-48db6ac66388", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91512025-04-30T06:30:12.059ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
91522025-04-30T06:30:12.061ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9f7346e-5cde-4e45-b771-bf623295f862", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91532025-04-30T06:30:12.061ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
91542025-04-30T06:30:12.061ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91552025-04-30T06:30:12.061ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91562025-04-30T06:30:12.061ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91572025-04-30T06:30:12.062ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91582025-04-30T06:30:12.068ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91592025-04-30T06:30:12.079ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
91602025-04-30T06:30:12.080ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
91612025-04-30T06:30:12.080ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91622025-04-30T06:30:12.080ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91632025-04-30T06:30:12.080ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91642025-04-30T06:30:12.081ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91652025-04-30T06:30:12.081ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91662025-04-30T06:30:12.085ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91672025-04-30T06:30:12.085ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91682025-04-30T06:30:12.085ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
91692025-04-30T06:30:12.085ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91702025-04-30T06:30:12.087ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91712025-04-30T06:30:12.094ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
91722025-04-30T06:30:12.094ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91732025-04-30T06:30:12.094ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91742025-04-30T06:30:12.094ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91752025-04-30T06:30:12.096ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91762025-04-30T06:30:12.106ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91772025-04-30T06:30:12.114ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91782025-04-30T06:30:12.114ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
91792025-04-30T06:30:12.115ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91802025-04-30T06:30:12.190ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
91812025-04-30T06:30:12.190ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
91822025-04-30T06:30:12.192ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:52992 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
91832025-04-30T06:30:12.193ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:52992 req_id = 580c4eb0-e155-4890-9a6f-8168d1ea4363 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
91842025-04-30T06:30:12.194ZINFOtest_image_promotion (dropshot): request completed latency_us = 1129 local_addr = [::1]:60070 method = GET remote_addr = [::1]:52992 req_id = 580c4eb0-e155-4890-9a6f-8168d1ea4363 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
91852025-04-30T06:30:12.194ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91862025-04-30T06:30:12.194ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
91872025-04-30T06:30:12.198ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91882025-04-30T06:30:12.199ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "580c4eb0-e155-4890-9a6f-8168d1ea4363", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:11 GMT"} })
91892025-04-30T06:30:12.201ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
91902025-04-30T06:30:12.201ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
91912025-04-30T06:30:12.201ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
91922025-04-30T06:30:12.203ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
91932025-04-30T06:30:12.203ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91942025-04-30T06:30:12.203ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91952025-04-30T06:30:12.203ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91962025-04-30T06:30:12.204ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91972025-04-30T06:30:12.209ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91982025-04-30T06:30:12.213ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
91992025-04-30T06:30:12.214ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92002025-04-30T06:30:12.214ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92012025-04-30T06:30:12.214ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92022025-04-30T06:30:12.281ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
92032025-04-30T06:30:12.282ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
92042025-04-30T06:30:12.282ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
92052025-04-30T06:30:12.282ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 513.725684ms iteration = 31 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
92062025-04-30T06:30:12.282ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 32 reason = Signaled
92072025-04-30T06:30:12.282ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92082025-04-30T06:30:12.282ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92092025-04-30T06:30:12.282ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92102025-04-30T06:30:12.282ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92112025-04-30T06:30:12.283ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92122025-04-30T06:30:12.291ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92132025-04-30T06:30:12.298ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92142025-04-30T06:30:12.298ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92152025-04-30T06:30:12.298ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92162025-04-30T06:30:12.299ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92172025-04-30T06:30:12.304ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
92182025-04-30T06:30:12.304ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
92192025-04-30T06:30:12.307ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 28 24 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ($..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57726 req_id = 92669144-744d-4094-b933-6d342a22e52a
92202025-04-30T06:30:12.307ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10276,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57726 req_id = 92669144-744d-4094-b933-6d342a22e52a
92212025-04-30T06:30:12.307ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
92222025-04-30T06:30:12.307ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
92232025-04-30T06:30:12.307ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57726 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 92669144-744d-4094-b933-6d342a22e52a
92242025-04-30T06:30:12.308ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3477, tv_nsec: 107072169 } })
92252025-04-30T06:30:12.308ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
92262025-04-30T06:30:12.309ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 65 28 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e e(..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63288 req_id = 966965dc-29ad-48e0-9edb-1cb01f706d3a
92272025-04-30T06:30:12.309ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25896,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63288 req_id = 966965dc-29ad-48e0-9edb-1cb01f706d3a
92282025-04-30T06:30:12.309ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
92292025-04-30T06:30:12.309ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63288 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 966965dc-29ad-48e0-9edb-1cb01f706d3a
92302025-04-30T06:30:12.373ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
92312025-04-30T06:30:12.376ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92322025-04-30T06:30:12.378ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92332025-04-30T06:30:12.378ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92342025-04-30T06:30:12.379ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
92352025-04-30T06:30:12.380ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92362025-04-30T06:30:12.381ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c2 0c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62003 req_id = 9ef1142c-8cd5-449c-b80c-cab25b6d7871
92372025-04-30T06:30:12.381ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49676,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62003 req_id = 9ef1142c-8cd5-449c-b80c-cab25b6d7871
92382025-04-30T06:30:12.381ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
92392025-04-30T06:30:12.381ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49676, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62003 req_id = 9ef1142c-8cd5-449c-b80c-cab25b6d7871
92402025-04-30T06:30:12.382ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
92412025-04-30T06:30:12.383ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
92422025-04-30T06:30:12.383ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
92432025-04-30T06:30:12.384ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92442025-04-30T06:30:12.390ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92452025-04-30T06:30:12.390ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92462025-04-30T06:30:12.390ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
92472025-04-30T06:30:12.391ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92482025-04-30T06:30:12.395ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a9f8d86-b94d-409d-8008-75131c760e46", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92492025-04-30T06:30:12.395ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
92502025-04-30T06:30:12.397ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92512025-04-30T06:30:12.397ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
92522025-04-30T06:30:12.398ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92532025-04-30T06:30:12.400ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dde8838-f23b-45a8-8dc5-51bba9d95785", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92542025-04-30T06:30:12.401ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
92552025-04-30T06:30:12.404ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51943ba6-bdfe-4a27-8fb8-214ae81be130", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92562025-04-30T06:30:12.404ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
92572025-04-30T06:30:12.411ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57fe8c6c-6fc9-45e0-b4d9-cc3ee2c654d8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92582025-04-30T06:30:12.412ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
92592025-04-30T06:30:12.420ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42293e6d-c410-4ddc-8730-18908b019205", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92602025-04-30T06:30:12.420ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
92612025-04-30T06:30:12.424ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb33f4ff-7e05-4d8e-9beb-094dafe509e4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92622025-04-30T06:30:12.424ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
92632025-04-30T06:30:12.435ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3d5efdc-343c-4201-8e0f-3952f19caebe", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92642025-04-30T06:30:12.437ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
92652025-04-30T06:30:12.441ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
92662025-04-30T06:30:12.446ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "976f8d34-71c2-43a5-a8bf-4b459af7b908", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92672025-04-30T06:30:12.446ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
92682025-04-30T06:30:12.447ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfbfa873-ccf3-49af-9fa3-818510f163c7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92692025-04-30T06:30:12.447ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
92702025-04-30T06:30:12.449ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b5ed1ff-2f93-45d6-bf32-c8fb1c3c2d8b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92712025-04-30T06:30:12.449ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
92722025-04-30T06:30:12.450ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e87b27f4-b762-4022-83e3-b39a6fe4244e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92732025-04-30T06:30:12.451ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
92742025-04-30T06:30:12.452ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6da281c9-5790-44aa-ab86-ac7dcbf78ebd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92752025-04-30T06:30:12.452ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
92762025-04-30T06:30:12.453ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af4a9606-ae5e-4e65-9957-1a3ecf195979", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92772025-04-30T06:30:12.453ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
92782025-04-30T06:30:12.455ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67b0416f-c7ed-471c-86c6-3a5e76a9f280", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92792025-04-30T06:30:12.455ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
92802025-04-30T06:30:12.457ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a65c38d-a90f-458b-bc1c-6b783710d3bd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92812025-04-30T06:30:12.457ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
92822025-04-30T06:30:12.459ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "533fd167-dc0b-4203-b6ab-7516a24e9e60", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92832025-04-30T06:30:12.459ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
92842025-04-30T06:30:12.462ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f88f244-b2aa-4804-9062-e8715af3f7a4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92852025-04-30T06:30:12.463ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
92862025-04-30T06:30:12.467ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7c9d6e1-1a9c-447a-bdb3-85938b9eb955", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92872025-04-30T06:30:12.467ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
92882025-04-30T06:30:12.469ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "943ee6d0-bef7-4ff7-8435-c68b9b405c86", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92892025-04-30T06:30:12.469ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
92902025-04-30T06:30:12.470ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c5173a4-dbae-4b81-9a53-fab787b0995d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92912025-04-30T06:30:12.472ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
92922025-04-30T06:30:12.472ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6420c5d2-df48-43c3-9f9e-b01a934a29e4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92932025-04-30T06:30:12.472ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
92942025-04-30T06:30:12.472ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e1e8a75-9b83-44c4-9d0d-a16db2f926a1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92952025-04-30T06:30:12.473ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
92962025-04-30T06:30:12.474ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cdc4522-a5ba-403d-bd7d-6cebde7f11d4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92972025-04-30T06:30:12.475ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
92982025-04-30T06:30:12.476ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "007fb131-c9fa-4238-822f-c2950119a83b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
92992025-04-30T06:30:12.476ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
93002025-04-30T06:30:12.478ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44c63849-0dfd-4c07-9ef9-845345df274a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93012025-04-30T06:30:12.479ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
93022025-04-30T06:30:12.481ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96998144-df8c-4564-9d77-ddfd1088239c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93032025-04-30T06:30:12.481ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
93042025-04-30T06:30:12.485ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95056e58-65c7-4e76-b243-efb3172b2be7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93052025-04-30T06:30:12.485ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
93062025-04-30T06:30:12.486ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd51fc35-5724-4f13-8b1f-35e391355186", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93072025-04-30T06:30:12.486ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
93082025-04-30T06:30:12.488ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30251b4f-7995-4aac-8538-3f89a8225df1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93092025-04-30T06:30:12.488ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
93102025-04-30T06:30:12.490ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00f48847-a711-41c7-aa94-cb0ce774f894", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93112025-04-30T06:30:12.490ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
93122025-04-30T06:30:12.491ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04b7729f-261b-467a-a323-a90ac305df0a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93132025-04-30T06:30:12.491ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
93142025-04-30T06:30:12.492ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93152025-04-30T06:30:12.492ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93162025-04-30T06:30:12.492ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93172025-04-30T06:30:12.492ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
93182025-04-30T06:30:12.493ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93192025-04-30T06:30:12.494ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef3135aa-2bbc-41cf-8f24-86da824ac479", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93202025-04-30T06:30:12.494ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
93212025-04-30T06:30:12.495ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48fc1a8c-242c-464c-835f-9fd5e2130b1d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93222025-04-30T06:30:12.496ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
93232025-04-30T06:30:12.496ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93242025-04-30T06:30:12.504ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
93252025-04-30T06:30:12.504ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93262025-04-30T06:30:12.504ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
93272025-04-30T06:30:12.505ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93282025-04-30T06:30:12.506ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
93292025-04-30T06:30:12.506ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93302025-04-30T06:30:12.508ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 880610fc-d097-45af-8a34-59881aeec641 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93312025-04-30T06:30:12.508ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 880610fc-d097-45af-8a34-59881aeec641 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93322025-04-30T06:30:12.508ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 880610fc-d097-45af-8a34-59881aeec641 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93332025-04-30T06:30:12.509ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 880610fc-d097-45af-8a34-59881aeec641 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93342025-04-30T06:30:12.509ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93352025-04-30T06:30:12.509ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
93362025-04-30T06:30:12.510ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93372025-04-30T06:30:12.510ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0a23fb0-a11b-4e5d-a30f-61fc53e99c89", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93382025-04-30T06:30:12.510ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
93392025-04-30T06:30:12.512ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "67bee02c-c5ff-45e1-8bf4-2987aa614db4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93402025-04-30T06:30:12.513ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
93412025-04-30T06:30:12.513ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 5415 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 880610fc-d097-45af-8a34-59881aeec641 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93422025-04-30T06:30:12.514ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "880610fc-d097-45af-8a34-59881aeec641", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93432025-04-30T06:30:12.514ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
93442025-04-30T06:30:12.515ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 0de39f51-cdf2-4b44-8d3c-5144bbacfb2d uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
93452025-04-30T06:30:12.516ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 0de39f51-cdf2-4b44-8d3c-5144bbacfb2d resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
93462025-04-30T06:30:12.516ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 0de39f51-cdf2-4b44-8d3c-5144bbacfb2d roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
93472025-04-30T06:30:12.516ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 0de39f51-cdf2-4b44-8d3c-5144bbacfb2d resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
93482025-04-30T06:30:12.520ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 4591 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 0de39f51-cdf2-4b44-8d3c-5144bbacfb2d response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
93492025-04-30T06:30:12.522ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "0de39f51-cdf2-4b44-8d3c-5144bbacfb2d", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93502025-04-30T06:30:12.522ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93512025-04-30T06:30:12.522ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
93522025-04-30T06:30:12.522ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
93532025-04-30T06:30:12.523ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7a4c4cd-c8e6-41a3-b90a-323def7c4a5c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93542025-04-30T06:30:12.523ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
93552025-04-30T06:30:12.526ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca4961de-e3f8-40d8-982f-2e442d94030b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93562025-04-30T06:30:12.526ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
93572025-04-30T06:30:12.528ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48534c06-10f9-4b72-8fcc-35d949ef0192", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93582025-04-30T06:30:12.528ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
93592025-04-30T06:30:12.530ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12a2e92b-a5dd-4719-90dd-7f64d474cde0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93602025-04-30T06:30:12.530ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
93612025-04-30T06:30:12.537ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1cb0149-def2-4978-bd45-e17f0d3036d6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93622025-04-30T06:30:12.537ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
93632025-04-30T06:30:12.540ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93642025-04-30T06:30:12.540ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93652025-04-30T06:30:12.541ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93662025-04-30T06:30:12.542ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93672025-04-30T06:30:12.542ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93682025-04-30T06:30:12.542ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93692025-04-30T06:30:12.548ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
93702025-04-30T06:30:12.550ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b896387-f1c3-4b44-9448-c66407f047c7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93712025-04-30T06:30:12.551ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
93722025-04-30T06:30:12.552ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b8fc533-2972-4b6e-a080-acc287c8be38", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93732025-04-30T06:30:12.552ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
93742025-04-30T06:30:12.553ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fa325c3-373c-4475-924a-64cbe7e5a334", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93752025-04-30T06:30:12.553ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
93762025-04-30T06:30:12.554ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a300b80e-fd3e-46b6-918f-17c177f2a9da", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93772025-04-30T06:30:12.554ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
93782025-04-30T06:30:12.555ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40fad422-ccae-4f95-a1ba-f160d2cab298", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
93792025-04-30T06:30:12.555ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
93802025-04-30T06:30:12.555ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93812025-04-30T06:30:12.555ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93822025-04-30T06:30:12.555ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93832025-04-30T06:30:12.557ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93842025-04-30T06:30:12.560ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93852025-04-30T06:30:12.568ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
93862025-04-30T06:30:12.568ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
93872025-04-30T06:30:12.568ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93882025-04-30T06:30:12.568ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93892025-04-30T06:30:12.568ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93902025-04-30T06:30:12.569ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93912025-04-30T06:30:12.575ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93922025-04-30T06:30:12.583ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
93932025-04-30T06:30:12.583ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93942025-04-30T06:30:12.583ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93952025-04-30T06:30:12.584ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93962025-04-30T06:30:12.667ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
93972025-04-30T06:30:12.667ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
93982025-04-30T06:30:12.667ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93992025-04-30T06:30:12.667ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94002025-04-30T06:30:12.667ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94012025-04-30T06:30:12.667ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
94022025-04-30T06:30:12.668ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94032025-04-30T06:30:12.668ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:55343 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
94042025-04-30T06:30:12.669ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:55343 req_id = 3a053796-f265-41fa-8f0e-a79a49730b63 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94052025-04-30T06:30:12.670ZINFOtest_image_promotion (dropshot): request completed latency_us = 476 local_addr = [::1]:60070 method = GET remote_addr = [::1]:55343 req_id = 3a053796-f265-41fa-8f0e-a79a49730b63 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94062025-04-30T06:30:12.671ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a053796-f265-41fa-8f0e-a79a49730b63", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
94072025-04-30T06:30:12.674ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
94082025-04-30T06:30:12.674ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
94092025-04-30T06:30:12.674ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
94102025-04-30T06:30:12.676ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
94112025-04-30T06:30:12.676ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94122025-04-30T06:30:12.676ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94132025-04-30T06:30:12.676ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94142025-04-30T06:30:12.676ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94152025-04-30T06:30:12.677ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94162025-04-30T06:30:12.687ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94172025-04-30T06:30:12.687ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94182025-04-30T06:30:12.687ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
94192025-04-30T06:30:12.687ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94202025-04-30T06:30:12.690ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94212025-04-30T06:30:12.694ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
94222025-04-30T06:30:12.694ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94232025-04-30T06:30:12.694ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94242025-04-30T06:30:12.694ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94252025-04-30T06:30:12.696ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94262025-04-30T06:30:12.697ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
94272025-04-30T06:30:12.697ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94282025-04-30T06:30:12.750ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
94292025-04-30T06:30:12.750ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
94302025-04-30T06:30:12.751ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
94312025-04-30T06:30:12.752ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
94322025-04-30T06:30:12.758ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
94332025-04-30T06:30:12.758ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
94342025-04-30T06:30:12.759ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
94352025-04-30T06:30:12.759ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
94362025-04-30T06:30:12.759ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 477.51603ms iteration = 32 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
94372025-04-30T06:30:12.759ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 33 reason = Signaled
94382025-04-30T06:30:12.759ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
94392025-04-30T06:30:12.760ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94402025-04-30T06:30:12.760ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94412025-04-30T06:30:12.760ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94422025-04-30T06:30:12.760ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94432025-04-30T06:30:12.766ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
94442025-04-30T06:30:12.766ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
94452025-04-30T06:30:12.767ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94462025-04-30T06:30:12.772ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
94472025-04-30T06:30:12.772ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94482025-04-30T06:30:12.772ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94492025-04-30T06:30:12.773ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94502025-04-30T06:30:12.776ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
94512025-04-30T06:30:12.776ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
94522025-04-30T06:30:12.777ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 75 43 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 uC..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49997 req_id = d0730cd1-f7f5-478d-aca8-9951c7dc4b59
94532025-04-30T06:30:12.777ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30019,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49997 req_id = d0730cd1-f7f5-478d-aca8-9951c7dc4b59
94542025-04-30T06:30:12.777ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
94552025-04-30T06:30:12.778ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
94562025-04-30T06:30:12.778ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49997 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d0730cd1-f7f5-478d-aca8-9951c7dc4b59
94572025-04-30T06:30:12.778ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
94582025-04-30T06:30:12.778ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3477, tv_nsec: 577788951 } })
94592025-04-30T06:30:12.778ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
94602025-04-30T06:30:12.780ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 47 e8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e G...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56241 req_id = fe8e74d3-25b7-4053-8f40-a6812b4b5605
94612025-04-30T06:30:12.780ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18408,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56241 req_id = fe8e74d3-25b7-4053-8f40-a6812b4b5605
94622025-04-30T06:30:12.780ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
94632025-04-30T06:30:12.781ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56241 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fe8e74d3-25b7-4053-8f40-a6812b4b5605
94642025-04-30T06:30:12.841ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
94652025-04-30T06:30:12.841ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94662025-04-30T06:30:12.841ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94672025-04-30T06:30:12.841ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94682025-04-30T06:30:12.841ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
94692025-04-30T06:30:12.842ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94702025-04-30T06:30:12.843ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: fa 4a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .J..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57238 req_id = 0447bdeb-fa78-423b-98c0-2b88c4d7890e
94712025-04-30T06:30:12.843ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64074,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57238 req_id = 0447bdeb-fa78-423b-98c0-2b88c4d7890e
94722025-04-30T06:30:12.843ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
94732025-04-30T06:30:12.843ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64074, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57238 req_id = 0447bdeb-fa78-423b-98c0-2b88c4d7890e
94742025-04-30T06:30:12.844ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
94752025-04-30T06:30:12.844ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
94762025-04-30T06:30:12.844ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
94772025-04-30T06:30:12.845ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94782025-04-30T06:30:12.851ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94792025-04-30T06:30:12.851ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94802025-04-30T06:30:12.852ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
94812025-04-30T06:30:12.852ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94822025-04-30T06:30:12.853ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
94832025-04-30T06:30:12.854ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "943a377a-48d9-4e15-869a-319f180e48a9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
94842025-04-30T06:30:12.855ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
94852025-04-30T06:30:12.855ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94862025-04-30T06:30:12.855ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
94872025-04-30T06:30:12.856ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94882025-04-30T06:30:12.864ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f893549-d47f-4263-b1bd-d1fb9ba23ebf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
94892025-04-30T06:30:12.864ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
94902025-04-30T06:30:12.869ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97c32dfb-ca23-40b3-9b89-312222e458db", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
94912025-04-30T06:30:12.870ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
94922025-04-30T06:30:12.872ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07344283-853f-41b3-b3b1-adf44764775d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
94932025-04-30T06:30:12.873ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
94942025-04-30T06:30:12.877ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d35c8abf-2a51-4572-aa16-b1ababc093aa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
94952025-04-30T06:30:12.877ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
94962025-04-30T06:30:12.880ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09b1532d-3a19-46c9-8bb1-d7eb2d31e358", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
94972025-04-30T06:30:12.880ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
94982025-04-30T06:30:12.882ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bdae26a-58e7-40c0-ac9b-95abfe6f9129", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
94992025-04-30T06:30:12.882ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
95002025-04-30T06:30:12.884ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a976142-dc00-4c84-bcdb-cbd403f3d13f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95012025-04-30T06:30:12.885ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
95022025-04-30T06:30:12.886ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90c87ebe-de89-4aa1-8b5a-8007abb45e89", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95032025-04-30T06:30:12.886ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
95042025-04-30T06:30:12.889ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
95052025-04-30T06:30:12.889ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad100d2e-30bd-4cce-ba55-858edf03a44c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95062025-04-30T06:30:12.889ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
95072025-04-30T06:30:12.891ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "479b3f32-65e2-4c09-93cd-4334958b2f7c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95082025-04-30T06:30:12.891ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
95092025-04-30T06:30:12.893ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b2d45cc-6417-4c77-bc17-e31417814e8e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95102025-04-30T06:30:12.893ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
95112025-04-30T06:30:12.894ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01bb7d08-767f-47ce-b27e-a7e6813c12a4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95122025-04-30T06:30:12.895ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
95132025-04-30T06:30:12.897ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ba9d2c1-f12c-48b8-ad17-e6a69cac3ba5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95142025-04-30T06:30:12.897ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
95152025-04-30T06:30:12.900ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60846c53-d4de-476f-b71a-7ce4e2dfa30e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95162025-04-30T06:30:12.900ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
95172025-04-30T06:30:12.902ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d45dd34-428c-4d89-98e0-62c74382ffa6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95182025-04-30T06:30:12.902ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
95192025-04-30T06:30:12.904ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7ac60ff-1d02-4fc3-b303-b5876f791fda", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95202025-04-30T06:30:12.904ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
95212025-04-30T06:30:12.908ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e5a7daa-5094-435a-95b5-83a008c5f20c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95222025-04-30T06:30:12.908ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
95232025-04-30T06:30:12.911ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4af47d03-869e-4a9d-a36a-6ff0d42f1b9f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95242025-04-30T06:30:12.911ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
95252025-04-30T06:30:12.913ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9407b79e-bf0e-401c-90f9-2995a33934c5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95262025-04-30T06:30:12.913ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
95272025-04-30T06:30:12.915ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82432849-11e4-4a6e-8194-8f774da7ab2d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95282025-04-30T06:30:12.916ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
95292025-04-30T06:30:12.917ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e4af74b-68ba-4dec-86d4-ff4bf931e014", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95302025-04-30T06:30:12.917ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
95312025-04-30T06:30:12.919ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a41565d7-b1ce-4c0d-b0df-7e0651b2d37b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95322025-04-30T06:30:12.919ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
95332025-04-30T06:30:12.920ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0586a9f7-73f9-47de-b861-4edd95705abf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95342025-04-30T06:30:12.921ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
95352025-04-30T06:30:12.922ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e2ce889-e55e-46c1-89ab-8ca3c8e753c0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95362025-04-30T06:30:12.923ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
95372025-04-30T06:30:12.925ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96e691a0-5035-4ff0-ad42-eec3c85a0578", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95382025-04-30T06:30:12.925ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
95392025-04-30T06:30:12.927ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b408d764-0929-4848-a74d-0159ccdb4635", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95402025-04-30T06:30:12.927ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
95412025-04-30T06:30:12.929ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7bed40a-1c68-46b2-ab75-68e8405ba43c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95422025-04-30T06:30:12.929ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
95432025-04-30T06:30:12.931ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b8d1ff9-7b0e-431a-9b29-0485173eeeda", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95442025-04-30T06:30:12.931ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
95452025-04-30T06:30:12.934ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8df37119-ae8d-4b84-893a-ad1b125d9f52", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95462025-04-30T06:30:12.934ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
95472025-04-30T06:30:12.935ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ed9f5ee-ea96-4042-9709-358f32ebf784", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95482025-04-30T06:30:12.935ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
95492025-04-30T06:30:12.936ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e96de673-f0e8-41f8-8c60-dffa03d6920f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95502025-04-30T06:30:12.937ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
95512025-04-30T06:30:12.939ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9380affa-27ad-492b-aa37-1e4a4a840a00", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95522025-04-30T06:30:12.939ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
95532025-04-30T06:30:12.939ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95542025-04-30T06:30:12.940ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95552025-04-30T06:30:12.940ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95562025-04-30T06:30:12.940ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
95572025-04-30T06:30:12.950ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95582025-04-30T06:30:12.954ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "47ebb630-cffe-42da-8825-9dbe70952365", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95592025-04-30T06:30:12.954ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
95602025-04-30T06:30:12.955ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95612025-04-30T06:30:12.961ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95622025-04-30T06:30:12.961ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95632025-04-30T06:30:12.962ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
95642025-04-30T06:30:12.962ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95652025-04-30T06:30:12.967ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95662025-04-30T06:30:12.968ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
95672025-04-30T06:30:12.968ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95682025-04-30T06:30:12.987ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdf503cf-eba1-4c4b-962b-43fd79a1dedf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95692025-04-30T06:30:12.990ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
95702025-04-30T06:30:13.025ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b71fb34-2487-4c96-9e99-7a476d441142", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95712025-04-30T06:30:13.026ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
95722025-04-30T06:30:13.032ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
95732025-04-30T06:30:13.035ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f50ddcbf-8761-46c1-b199-9f5656ed1e63", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95742025-04-30T06:30:13.035ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
95752025-04-30T06:30:13.041ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "240a053c-eb14-4b28-aee4-da7cbaf2e423", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95762025-04-30T06:30:13.041ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
95772025-04-30T06:30:13.048ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f3bb408-e3c7-4f57-a353-6d84c846c234", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95782025-04-30T06:30:13.048ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
95792025-04-30T06:30:13.053ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bb35e74-b76a-40f7-afa2-4940851e1b37", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95802025-04-30T06:30:13.053ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
95812025-04-30T06:30:13.060ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e01f15f-6d08-4ef0-a98b-5ebd5d9ea621", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95822025-04-30T06:30:13.061ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
95832025-04-30T06:30:13.062ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01d11d4c-1b43-4c53-9704-9d9b087648cb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95842025-04-30T06:30:13.063ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
95852025-04-30T06:30:13.064ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7427276f-4f26-453d-823a-d8107068a479", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95862025-04-30T06:30:13.065ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
95872025-04-30T06:30:13.066ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "456bcba9-005a-43ef-855e-5b5581c11413", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95882025-04-30T06:30:13.066ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
95892025-04-30T06:30:13.068ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24e2ead4-3e17-4f29-bf45-d0ab410dc93d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
95902025-04-30T06:30:13.068ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
95912025-04-30T06:30:13.069ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95922025-04-30T06:30:13.069ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95932025-04-30T06:30:13.069ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95942025-04-30T06:30:13.070ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95952025-04-30T06:30:13.074ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95962025-04-30T06:30:13.080ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
95972025-04-30T06:30:13.080ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
95982025-04-30T06:30:13.080ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95992025-04-30T06:30:13.080ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96002025-04-30T06:30:13.080ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96012025-04-30T06:30:13.081ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96022025-04-30T06:30:13.086ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96032025-04-30T06:30:13.086ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96042025-04-30T06:30:13.086ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96052025-04-30T06:30:13.086ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
96062025-04-30T06:30:13.087ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96072025-04-30T06:30:13.087ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96082025-04-30T06:30:13.111ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
96092025-04-30T06:30:13.111ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96102025-04-30T06:30:13.111ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96112025-04-30T06:30:13.112ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96122025-04-30T06:30:13.117ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96132025-04-30T06:30:13.131ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96142025-04-30T06:30:13.131ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96152025-04-30T06:30:13.131ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
96162025-04-30T06:30:13.131ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96172025-04-30T06:30:13.215ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
96182025-04-30T06:30:13.216ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
96192025-04-30T06:30:13.217ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:56709 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
96202025-04-30T06:30:13.218ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96212025-04-30T06:30:13.218ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
96222025-04-30T06:30:13.219ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96232025-04-30T06:30:13.219ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:56709 req_id = 7ffa4412-ac7c-4022-8039-a2a80bbdfaab sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
96242025-04-30T06:30:13.221ZINFOtest_image_promotion (dropshot): request completed latency_us = 1692 local_addr = [::1]:60070 method = GET remote_addr = [::1]:56709 req_id = 7ffa4412-ac7c-4022-8039-a2a80bbdfaab response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
96252025-04-30T06:30:13.221ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ffa4412-ac7c-4022-8039-a2a80bbdfaab", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:12 GMT"} })
96262025-04-30T06:30:13.224ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
96272025-04-30T06:30:13.224ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
96282025-04-30T06:30:13.224ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
96292025-04-30T06:30:13.226ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
96302025-04-30T06:30:13.226ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96312025-04-30T06:30:13.226ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96322025-04-30T06:30:13.227ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96332025-04-30T06:30:13.227ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96342025-04-30T06:30:13.231ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96352025-04-30T06:30:13.248ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
96362025-04-30T06:30:13.249ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96372025-04-30T06:30:13.249ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96382025-04-30T06:30:13.249ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96392025-04-30T06:30:13.286ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
96402025-04-30T06:30:13.288ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
96412025-04-30T06:30:13.289ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
96422025-04-30T06:30:13.289ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 529.159345ms iteration = 33 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
96432025-04-30T06:30:13.289ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 34 reason = Signaled
96442025-04-30T06:30:13.289ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
96452025-04-30T06:30:13.289ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96462025-04-30T06:30:13.289ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96472025-04-30T06:30:13.289ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96482025-04-30T06:30:13.290ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96492025-04-30T06:30:13.293ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96502025-04-30T06:30:13.299ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
96512025-04-30T06:30:13.300ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96522025-04-30T06:30:13.300ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96532025-04-30T06:30:13.300ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96542025-04-30T06:30:13.317ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
96552025-04-30T06:30:13.317ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
96562025-04-30T06:30:13.319ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f7 50 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .P..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55949 req_id = 326520a3-c077-48de-98fa-e8be0cc52b70
96572025-04-30T06:30:13.319ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63312,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55949 req_id = 326520a3-c077-48de-98fa-e8be0cc52b70
96582025-04-30T06:30:13.336ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
96592025-04-30T06:30:13.336ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
96602025-04-30T06:30:13.336ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55949 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 326520a3-c077-48de-98fa-e8be0cc52b70
96612025-04-30T06:30:13.337ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96622025-04-30T06:30:13.337ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96632025-04-30T06:30:13.337ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96642025-04-30T06:30:13.337ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
96652025-04-30T06:30:13.339ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96662025-04-30T06:30:13.339ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3478, tv_nsec: 138709819 } })
96672025-04-30T06:30:13.340ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
96682025-04-30T06:30:13.342ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f3 8f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58988 req_id = dbecca21-dc19-4c3a-94a7-28636893a2c4
96692025-04-30T06:30:13.343ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62351,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58988 req_id = dbecca21-dc19-4c3a-94a7-28636893a2c4
96702025-04-30T06:30:13.343ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
96712025-04-30T06:30:13.343ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58988 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dbecca21-dc19-4c3a-94a7-28636893a2c4
96722025-04-30T06:30:13.412ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
96732025-04-30T06:30:13.413ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96742025-04-30T06:30:13.421ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96752025-04-30T06:30:13.421ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96762025-04-30T06:30:13.422ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
96772025-04-30T06:30:13.422ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96782025-04-30T06:30:13.423ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 58 e2 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c X............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58635 req_id = eeade654-3af0-4c04-955f-d78860edf060
96792025-04-30T06:30:13.423ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22754,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58635 req_id = eeade654-3af0-4c04-955f-d78860edf060
96802025-04-30T06:30:13.423ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
96812025-04-30T06:30:13.423ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22754, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58635 req_id = eeade654-3af0-4c04-955f-d78860edf060
96822025-04-30T06:30:13.423ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
96832025-04-30T06:30:13.423ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
96842025-04-30T06:30:13.423ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
96852025-04-30T06:30:13.425ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96862025-04-30T06:30:13.425ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
96872025-04-30T06:30:13.426ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96882025-04-30T06:30:13.426ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba3373b6-fd68-4fa2-9e94-bff84b6434d9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
96892025-04-30T06:30:13.426ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
96902025-04-30T06:30:13.428ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad11b96e-07f1-4b5f-a1cb-a7f1422f3c07", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
96912025-04-30T06:30:13.428ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
96922025-04-30T06:30:13.431ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98eb938b-fc05-41de-86a7-66fdf5cd3099", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
96932025-04-30T06:30:13.431ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
96942025-04-30T06:30:13.434ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "705f2ecb-39b3-4a12-a671-3d73f2bc532f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
96952025-04-30T06:30:13.434ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
96962025-04-30T06:30:13.436ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7886558-5054-479a-a4bf-b9cd4330e6fc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
96972025-04-30T06:30:13.436ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
96982025-04-30T06:30:13.441ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9102a95e-c443-4038-8eda-da1ed580dee9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
96992025-04-30T06:30:13.441ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
97002025-04-30T06:30:13.444ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e439de89-e9ff-422a-9882-e2b13e1d08db", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97012025-04-30T06:30:13.445ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
97022025-04-30T06:30:13.447ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf5f68b3-b457-4d10-a654-6d6baa5c0c51", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97032025-04-30T06:30:13.447ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
97042025-04-30T06:30:13.448ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e1f7794-b7fd-48a5-9526-d3919f586610", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97052025-04-30T06:30:13.448ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
97062025-04-30T06:30:13.450ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2162d925-a773-432f-a4aa-ede1a940624f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97072025-04-30T06:30:13.450ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
97082025-04-30T06:30:13.451ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa11af06-196d-454e-bacb-0feb1416d142", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97092025-04-30T06:30:13.451ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
97102025-04-30T06:30:13.452ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98e127a3-ec22-4b42-ba18-dc53fc45ef4e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97112025-04-30T06:30:13.452ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
97122025-04-30T06:30:13.454ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9cab7a1-14dc-4588-a829-8f2186c9e7ad", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97132025-04-30T06:30:13.454ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
97142025-04-30T06:30:13.459ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
97152025-04-30T06:30:13.459ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4052e289-83af-4f51-868a-0965e55d0a3c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97162025-04-30T06:30:13.460ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
97172025-04-30T06:30:13.463ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64d5d595-8491-44b7-9a1e-4c8c6a31a78c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97182025-04-30T06:30:13.463ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
97192025-04-30T06:30:13.465ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51bc48d6-b43a-4947-8bee-510624d5236d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97202025-04-30T06:30:13.466ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
97212025-04-30T06:30:13.469ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8357e9e-d7b2-47df-bee8-4abffdd6a2c3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97222025-04-30T06:30:13.470ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
97232025-04-30T06:30:13.474ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e34a698-2bbc-4aab-9c68-22353bf9eb5a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97242025-04-30T06:30:13.474ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
97252025-04-30T06:30:13.478ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51ec6427-0fec-419b-a34f-dbf2c677a6f7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97262025-04-30T06:30:13.479ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
97272025-04-30T06:30:13.480ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb7a8ea1-0c85-4abb-afe8-2251110a7955", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97282025-04-30T06:30:13.481ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
97292025-04-30T06:30:13.482ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c75b197-dabc-47f0-bd86-16deb112b1b7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97302025-04-30T06:30:13.482ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
97312025-04-30T06:30:13.484ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "633f30a7-b2a2-4c0d-b077-30f3bd818e1d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97322025-04-30T06:30:13.484ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
97332025-04-30T06:30:13.486ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d9ae87d-3400-4c60-aa9b-342bdf1b3371", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97342025-04-30T06:30:13.486ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
97352025-04-30T06:30:13.488ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b37174c7-14f8-444c-89f0-7fa87d1c9559", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97362025-04-30T06:30:13.488ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
97372025-04-30T06:30:13.491ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7197c729-20a2-494b-a7f9-37e751cc3902", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97382025-04-30T06:30:13.492ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
97392025-04-30T06:30:13.495ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "783e1c4e-d528-4b94-ad27-8b4f5752950a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97402025-04-30T06:30:13.495ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
97412025-04-30T06:30:13.498ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90b4a8f2-61ce-4fb3-b28f-14d919004d2c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97422025-04-30T06:30:13.498ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
97432025-04-30T06:30:13.510ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97442025-04-30T06:30:13.510ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97452025-04-30T06:30:13.510ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97462025-04-30T06:30:13.510ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
97472025-04-30T06:30:13.511ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97482025-04-30T06:30:13.512ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c27cd11-0e0a-49a4-9217-348c14060fcd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97492025-04-30T06:30:13.512ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
97502025-04-30T06:30:13.513ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97512025-04-30T06:30:13.522ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97522025-04-30T06:30:13.522ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97532025-04-30T06:30:13.522ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
97542025-04-30T06:30:13.523ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97552025-04-30T06:30:13.524ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a126c49-de08-4d0e-984d-1e6249685995", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97562025-04-30T06:30:13.526ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
97572025-04-30T06:30:13.526ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97582025-04-30T06:30:13.526ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
97592025-04-30T06:30:13.527ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97602025-04-30T06:30:13.529ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48940bfc-41df-47c1-beec-10e7fa7c2723", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97612025-04-30T06:30:13.530ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
97622025-04-30T06:30:13.532ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "350a0fdd-dcd0-4e39-835c-2779d10f2759", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97632025-04-30T06:30:13.532ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
97642025-04-30T06:30:13.533ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70af7bc0-1697-4569-9f57-44285ab11942", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97652025-04-30T06:30:13.533ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
97662025-04-30T06:30:13.537ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d5097c1-9592-413b-871c-76ff3a49a978", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97672025-04-30T06:30:13.538ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
97682025-04-30T06:30:13.538ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97692025-04-30T06:30:13.539ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97702025-04-30T06:30:13.541ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97712025-04-30T06:30:13.542ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97722025-04-30T06:30:13.542ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97732025-04-30T06:30:13.542ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97742025-04-30T06:30:13.543ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f87ae139-19f2-4733-90bf-97f8561ce7a6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97752025-04-30T06:30:13.543ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
97762025-04-30T06:30:13.544ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "250ab601-6998-42fb-8812-3b920892addf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97772025-04-30T06:30:13.545ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
97782025-04-30T06:30:13.552ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69ec1578-397f-4799-812d-797c8dd24b84", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97792025-04-30T06:30:13.552ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
97802025-04-30T06:30:13.558ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a5612cc-b112-44d1-a2c9-da24aeb10856", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97812025-04-30T06:30:13.558ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
97822025-04-30T06:30:13.564ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
97832025-04-30T06:30:13.565ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54de601f-6c51-4e30-9a6b-ef327abdc6dd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97842025-04-30T06:30:13.567ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
97852025-04-30T06:30:13.572ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d69c4766-eaeb-42ca-92ee-307e93382dfe", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97862025-04-30T06:30:13.572ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
97872025-04-30T06:30:13.577ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "624c3b21-bd7c-4267-a02f-0190a9f2980d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97882025-04-30T06:30:13.577ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
97892025-04-30T06:30:13.583ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2354660-b6f9-4c69-8357-12fa85625bf2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97902025-04-30T06:30:13.583ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
97912025-04-30T06:30:13.585ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2486e9d1-4a2b-4953-bd6c-029c6a591432", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97922025-04-30T06:30:13.585ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
97932025-04-30T06:30:13.586ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffb3008e-2d26-4d5d-9094-d59a0bf00372", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97942025-04-30T06:30:13.587ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
97952025-04-30T06:30:13.588ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d315ae8f-a853-4908-b369-58756a39f3ac", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97962025-04-30T06:30:13.588ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
97972025-04-30T06:30:13.589ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65cf9c9b-a165-4121-baf4-c4260eff16bc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
97982025-04-30T06:30:13.589ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
97992025-04-30T06:30:13.589ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98002025-04-30T06:30:13.589ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98012025-04-30T06:30:13.589ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98022025-04-30T06:30:13.592ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98032025-04-30T06:30:13.598ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98042025-04-30T06:30:13.606ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
98052025-04-30T06:30:13.606ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
98062025-04-30T06:30:13.606ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98072025-04-30T06:30:13.606ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98082025-04-30T06:30:13.606ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98092025-04-30T06:30:13.607ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98102025-04-30T06:30:13.629ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98112025-04-30T06:30:13.629ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98122025-04-30T06:30:13.629ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98132025-04-30T06:30:13.630ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
98142025-04-30T06:30:13.633ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98152025-04-30T06:30:13.634ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98162025-04-30T06:30:13.643ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
98172025-04-30T06:30:13.643ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98182025-04-30T06:30:13.643ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98192025-04-30T06:30:13.644ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98202025-04-30T06:30:13.648ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98212025-04-30T06:30:13.661ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98222025-04-30T06:30:13.661ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98232025-04-30T06:30:13.661ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
98242025-04-30T06:30:13.661ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98252025-04-30T06:30:13.715ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
98262025-04-30T06:30:13.716ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
98272025-04-30T06:30:13.718ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:56299 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
98282025-04-30T06:30:13.719ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:56299 req_id = 43bf7191-39b4-418d-a9e2-d7f9827ba6f4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
98292025-04-30T06:30:13.720ZINFOtest_image_promotion (dropshot): request completed latency_us = 568 local_addr = [::1]:60070 method = GET remote_addr = [::1]:56299 req_id = 43bf7191-39b4-418d-a9e2-d7f9827ba6f4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
98302025-04-30T06:30:13.720ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98312025-04-30T06:30:13.720ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
98322025-04-30T06:30:13.722ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98332025-04-30T06:30:13.722ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "43bf7191-39b4-418d-a9e2-d7f9827ba6f4", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
98342025-04-30T06:30:13.725ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
98352025-04-30T06:30:13.725ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
98362025-04-30T06:30:13.726ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
98372025-04-30T06:30:13.728ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
98382025-04-30T06:30:13.728ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98392025-04-30T06:30:13.728ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98402025-04-30T06:30:13.728ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98412025-04-30T06:30:13.729ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98422025-04-30T06:30:13.735ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98432025-04-30T06:30:13.743ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
98442025-04-30T06:30:13.744ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98452025-04-30T06:30:13.744ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98462025-04-30T06:30:13.745ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98472025-04-30T06:30:13.754ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
98482025-04-30T06:30:13.754ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
98492025-04-30T06:30:13.756ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
98502025-04-30T06:30:13.757ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
98512025-04-30T06:30:13.769ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
98522025-04-30T06:30:13.769ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
98532025-04-30T06:30:13.782ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
98542025-04-30T06:30:13.782ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
98552025-04-30T06:30:13.793ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
98562025-04-30T06:30:13.797ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
98572025-04-30T06:30:13.797ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
98582025-04-30T06:30:13.797ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 507.418839ms iteration = 34 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
98592025-04-30T06:30:13.797ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 35 reason = Signaled
98602025-04-30T06:30:13.797ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
98612025-04-30T06:30:13.797ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98622025-04-30T06:30:13.797ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98632025-04-30T06:30:13.797ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98642025-04-30T06:30:13.797ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98652025-04-30T06:30:13.801ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98662025-04-30T06:30:13.805ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
98672025-04-30T06:30:13.805ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98682025-04-30T06:30:13.805ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98692025-04-30T06:30:13.805ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98702025-04-30T06:30:13.809ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
98712025-04-30T06:30:13.810ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
98722025-04-30T06:30:13.811ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6e 07 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 n............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55626 req_id = efe07f7d-c75c-4e54-88a7-7875233c1785
98732025-04-30T06:30:13.811ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28167,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55626 req_id = efe07f7d-c75c-4e54-88a7-7875233c1785
98742025-04-30T06:30:13.813ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
98752025-04-30T06:30:13.813ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
98762025-04-30T06:30:13.813ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55626 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = efe07f7d-c75c-4e54-88a7-7875233c1785
98772025-04-30T06:30:13.814ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3478, tv_nsec: 612743670 } })
98782025-04-30T06:30:13.814ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
98792025-04-30T06:30:13.814ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 44 cc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e D...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64308 req_id = 4f9ae005-4e9e-495d-9874-92c3abb057f1
98802025-04-30T06:30:13.814ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17612,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64308 req_id = 4f9ae005-4e9e-495d-9874-92c3abb057f1
98812025-04-30T06:30:13.814ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
98822025-04-30T06:30:13.815ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64308 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4f9ae005-4e9e-495d-9874-92c3abb057f1
98832025-04-30T06:30:13.864ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
98842025-04-30T06:30:13.865ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98852025-04-30T06:30:13.865ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98862025-04-30T06:30:13.865ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98872025-04-30T06:30:13.865ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
98882025-04-30T06:30:13.866ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98892025-04-30T06:30:13.866ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
98902025-04-30T06:30:13.867ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9f 4e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .N..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54847 req_id = 7ea0b328-1580-4043-bff0-48e77e67d180
98912025-04-30T06:30:13.867ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40782,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54847 req_id = 7ea0b328-1580-4043-bff0-48e77e67d180
98922025-04-30T06:30:13.867ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
98932025-04-30T06:30:13.868ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40782, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54847 req_id = 7ea0b328-1580-4043-bff0-48e77e67d180
98942025-04-30T06:30:13.868ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
98952025-04-30T06:30:13.868ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
98962025-04-30T06:30:13.868ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
98972025-04-30T06:30:13.869ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98982025-04-30T06:30:13.874ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98992025-04-30T06:30:13.874ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99002025-04-30T06:30:13.874ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
99012025-04-30T06:30:13.875ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99022025-04-30T06:30:13.876ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "253b318c-d11c-40ca-b0c5-25aa2817679d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99032025-04-30T06:30:13.877ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
99042025-04-30T06:30:13.879ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99052025-04-30T06:30:13.879ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
99062025-04-30T06:30:13.881ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99072025-04-30T06:30:13.882ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f461a61d-24ce-4ef9-8bc7-4f7c2a35fb69", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99082025-04-30T06:30:13.882ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
99092025-04-30T06:30:13.886ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be61cb50-8b80-4c70-ac48-38320cddd52f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99102025-04-30T06:30:13.886ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
99112025-04-30T06:30:13.891ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecd98799-b587-4a33-9c43-21c5e0f09a92", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99122025-04-30T06:30:13.891ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
99132025-04-30T06:30:13.897ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9231f87-9084-4609-b7d4-3f2d4666c562", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99142025-04-30T06:30:13.897ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
99152025-04-30T06:30:13.904ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ac156fd-6cd6-44dc-afea-6e56e3c8bb04", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99162025-04-30T06:30:13.904ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
99172025-04-30T06:30:13.915ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e4cc6af-8e31-470e-95eb-593db7c58584", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99182025-04-30T06:30:13.915ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
99192025-04-30T06:30:13.916ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
99202025-04-30T06:30:13.923ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06e1c5dc-d5c3-4172-b0ab-d35b47d8b3a3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99212025-04-30T06:30:13.923ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
99222025-04-30T06:30:13.926ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65278acd-0373-4364-bc4c-1ff58e63c62b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99232025-04-30T06:30:13.926ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
99242025-04-30T06:30:13.928ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66eb41eb-ce3e-4e16-a412-91477fb46f3f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99252025-04-30T06:30:13.929ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
99262025-04-30T06:30:13.930ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fddb7ea-1f48-4910-94ee-bfadd9b5463b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99272025-04-30T06:30:13.930ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
99282025-04-30T06:30:13.931ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "196d5a39-4110-4ee5-ab3c-9f706070ec51", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99292025-04-30T06:30:13.931ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
99302025-04-30T06:30:13.933ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bc76092-14b2-4552-8fdc-4c419a312705", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99312025-04-30T06:30:13.933ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
99322025-04-30T06:30:13.935ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfc76320-0e50-4bb8-917c-5e5f334e5e1b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99332025-04-30T06:30:13.936ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
99342025-04-30T06:30:13.940ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc95c872-c4d9-45a8-a875-1efcadf8058b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99352025-04-30T06:30:13.940ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
99362025-04-30T06:30:13.942ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cab4046a-ae46-4261-8ce7-1e670a2953e9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99372025-04-30T06:30:13.943ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
99382025-04-30T06:30:13.946ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ad18ed8-6584-4a6c-bbca-e4a7ea70e525", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99392025-04-30T06:30:13.946ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
99402025-04-30T06:30:13.968ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99412025-04-30T06:30:13.968ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99422025-04-30T06:30:13.968ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99432025-04-30T06:30:13.968ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
99442025-04-30T06:30:13.969ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99452025-04-30T06:30:13.980ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e61ae48-fe10-4155-bc1c-4c13ecd05f86", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99462025-04-30T06:30:13.980ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
99472025-04-30T06:30:13.981ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99482025-04-30T06:30:13.986ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99492025-04-30T06:30:13.986ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99502025-04-30T06:30:13.986ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
99512025-04-30T06:30:13.987ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99522025-04-30T06:30:13.993ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99532025-04-30T06:30:13.993ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
99542025-04-30T06:30:13.996ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99552025-04-30T06:30:14.021ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4246bb93-5faa-468f-be40-04f3acb5e11a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99562025-04-30T06:30:14.021ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
99572025-04-30T06:30:14.023ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "621f5445-442e-4f7a-81c3-ff4714301fb0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99582025-04-30T06:30:14.023ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
99592025-04-30T06:30:14.026ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4411235b-8aa9-4be4-9cd6-14334b7c3bad", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99602025-04-30T06:30:14.028ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
99612025-04-30T06:30:14.029ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "589233c6-067f-4ed9-8d95-62dabc7c6e79", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99622025-04-30T06:30:14.029ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
99632025-04-30T06:30:14.032ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05837cca-38ff-46a4-a545-2d43e7063031", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99642025-04-30T06:30:14.032ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
99652025-04-30T06:30:14.034ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d8d1f70-948c-4d44-8082-a39bd1e1f029", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99662025-04-30T06:30:14.036ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
99672025-04-30T06:30:14.047ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3795565c-1058-40af-a042-e397e02739e5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99682025-04-30T06:30:14.047ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
99692025-04-30T06:30:14.053ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
99702025-04-30T06:30:14.055ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "616c7c4e-4258-4049-925d-9325bf7c73ef", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99712025-04-30T06:30:14.055ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
99722025-04-30T06:30:14.057ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97e15044-0116-41f4-8b4f-75e7abe9ee3f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99732025-04-30T06:30:14.057ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
99742025-04-30T06:30:14.060ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8964769a-b096-4154-a77b-8135d374c186", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99752025-04-30T06:30:14.060ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
99762025-04-30T06:30:14.062ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a618833-c759-454c-ba38-aed9e3c484e8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99772025-04-30T06:30:14.062ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
99782025-04-30T06:30:14.064ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "956930ff-cf1d-4a7a-abc5-d96a20328b41", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99792025-04-30T06:30:14.064ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
99802025-04-30T06:30:14.065ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91fe3eba-60d8-4306-8fb7-1ca6a5f036f5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99812025-04-30T06:30:14.066ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
99822025-04-30T06:30:14.067ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af67727d-d77d-4fd6-8d45-4a0b97aa5f86", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99832025-04-30T06:30:14.067ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
99842025-04-30T06:30:14.068ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f2524af-b288-407a-bc77-74263187ca8a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99852025-04-30T06:30:14.068ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
99862025-04-30T06:30:14.069ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5831dc9e-a369-4b6b-9f25-f2f8ea1d31ff", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99872025-04-30T06:30:14.070ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
99882025-04-30T06:30:14.071ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6b1285b-3f1b-4ac0-937a-ca5bd16a0fff", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99892025-04-30T06:30:14.071ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
99902025-04-30T06:30:14.073ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c51e4b74-850f-441c-890b-fc43a0ced7f6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99912025-04-30T06:30:14.073ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
99922025-04-30T06:30:14.076ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d1f4a28-0cab-4915-843f-3691e99f4b8c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99932025-04-30T06:30:14.076ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
99942025-04-30T06:30:14.078ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b394b9cc-ce63-4b5d-89f0-b164f101b2c1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99952025-04-30T06:30:14.078ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
99962025-04-30T06:30:14.082ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0371a84f-d21a-4d10-81b2-d4108c4f6412", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99972025-04-30T06:30:14.083ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
99982025-04-30T06:30:14.085ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ee72059-fc39-4f28-8f28-62573d404453", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
99992025-04-30T06:30:14.085ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
100002025-04-30T06:30:14.089ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9446775-989f-402a-904a-6b84d501ba9e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
100012025-04-30T06:30:14.089ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
100022025-04-30T06:30:14.091ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60b9c6a9-08b9-4066-845e-4a508ddc8f59", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
100032025-04-30T06:30:14.091ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
100042025-04-30T06:30:14.092ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e935884-8ab6-4ff1-a5de-fdc36a6d6800", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
100052025-04-30T06:30:14.092ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
100062025-04-30T06:30:14.093ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa82eb00-b9b9-4df5-ab8b-a2115d8b8714", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
100072025-04-30T06:30:14.093ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
100082025-04-30T06:30:14.094ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b7ee430-dd2f-4edf-a296-e3f20c70234d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
100092025-04-30T06:30:14.094ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
100102025-04-30T06:30:14.094ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100112025-04-30T06:30:14.094ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100122025-04-30T06:30:14.094ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100132025-04-30T06:30:14.095ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100142025-04-30T06:30:14.098ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100152025-04-30T06:30:14.105ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
100162025-04-30T06:30:14.105ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
100172025-04-30T06:30:14.105ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100182025-04-30T06:30:14.105ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100192025-04-30T06:30:14.105ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100202025-04-30T06:30:14.106ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100212025-04-30T06:30:14.107ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100222025-04-30T06:30:14.107ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100232025-04-30T06:30:14.107ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100242025-04-30T06:30:14.107ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
100252025-04-30T06:30:14.108ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100262025-04-30T06:30:14.110ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100272025-04-30T06:30:14.115ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
100282025-04-30T06:30:14.115ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100292025-04-30T06:30:14.115ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100302025-04-30T06:30:14.116ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100312025-04-30T06:30:14.119ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100322025-04-30T06:30:14.129ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100332025-04-30T06:30:14.129ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100342025-04-30T06:30:14.130ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
100352025-04-30T06:30:14.131ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100362025-04-30T06:30:14.222ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
100372025-04-30T06:30:14.222ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
100382025-04-30T06:30:14.223ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:38306 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
100392025-04-30T06:30:14.223ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:38306 req_id = 6ac3dc73-12f0-47e2-b436-9435a9df9f63 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
100402025-04-30T06:30:14.224ZINFOtest_image_promotion (dropshot): request completed latency_us = 457 local_addr = [::1]:60070 method = GET remote_addr = [::1]:38306 req_id = 6ac3dc73-12f0-47e2-b436-9435a9df9f63 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
100412025-04-30T06:30:14.224ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100422025-04-30T06:30:14.224ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
100432025-04-30T06:30:14.225ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100442025-04-30T06:30:14.225ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ac3dc73-12f0-47e2-b436-9435a9df9f63", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:13 GMT"} })
100452025-04-30T06:30:14.227ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
100462025-04-30T06:30:14.227ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
100472025-04-30T06:30:14.227ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
100482025-04-30T06:30:14.229ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
100492025-04-30T06:30:14.229ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100502025-04-30T06:30:14.229ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100512025-04-30T06:30:14.229ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100522025-04-30T06:30:14.229ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100532025-04-30T06:30:14.234ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100542025-04-30T06:30:14.238ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
100552025-04-30T06:30:14.239ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100562025-04-30T06:30:14.239ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100572025-04-30T06:30:14.239ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100582025-04-30T06:30:14.325ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
100592025-04-30T06:30:14.329ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
100602025-04-30T06:30:14.332ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
100612025-04-30T06:30:14.332ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 535.568576ms iteration = 35 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
100622025-04-30T06:30:14.332ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 36 reason = Signaled
100632025-04-30T06:30:14.332ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
100642025-04-30T06:30:14.332ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100652025-04-30T06:30:14.332ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100662025-04-30T06:30:14.332ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100672025-04-30T06:30:14.333ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100682025-04-30T06:30:14.367ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100692025-04-30T06:30:14.374ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
100702025-04-30T06:30:14.382ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100712025-04-30T06:30:14.385ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100722025-04-30T06:30:14.386ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100732025-04-30T06:30:14.387ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100742025-04-30T06:30:14.387ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100752025-04-30T06:30:14.387ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100762025-04-30T06:30:14.387ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
100772025-04-30T06:30:14.388ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100782025-04-30T06:30:14.394ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
100792025-04-30T06:30:14.394ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
100802025-04-30T06:30:14.395ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 7d b0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 }............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61594 req_id = 01566a1d-55d5-45c3-82e1-3070bba81e8c
100812025-04-30T06:30:14.395ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32176,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61594 req_id = 01566a1d-55d5-45c3-82e1-3070bba81e8c
100822025-04-30T06:30:14.396ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
100832025-04-30T06:30:14.397ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
100842025-04-30T06:30:14.397ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61594 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 01566a1d-55d5-45c3-82e1-3070bba81e8c
100852025-04-30T06:30:14.398ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100862025-04-30T06:30:14.408ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100872025-04-30T06:30:14.408ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100882025-04-30T06:30:14.408ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
100892025-04-30T06:30:14.409ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100902025-04-30T06:30:14.409ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3479, tv_nsec: 208275208 } })
100912025-04-30T06:30:14.409ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
100922025-04-30T06:30:14.410ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9d 3b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .;..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62091 req_id = e132ff8f-3260-4254-9c4b-a177f5bd81b3
100932025-04-30T06:30:14.410ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40251,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62091 req_id = e132ff8f-3260-4254-9c4b-a177f5bd81b3
100942025-04-30T06:30:14.411ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
100952025-04-30T06:30:14.439ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62091 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e132ff8f-3260-4254-9c4b-a177f5bd81b3
100962025-04-30T06:30:14.554ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
100972025-04-30T06:30:14.554ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100982025-04-30T06:30:14.554ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
100992025-04-30T06:30:14.556ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101002025-04-30T06:30:14.556ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
101012025-04-30T06:30:14.556ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101022025-04-30T06:30:14.556ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101032025-04-30T06:30:14.557ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101042025-04-30T06:30:14.559ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 91de8ef0-95ad-44df-9109-3223e1261929 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101052025-04-30T06:30:14.559ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 16 74 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .t..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63162 req_id = 6b025f42-2bd8-44c7-8a05-7fd1487e4238
101062025-04-30T06:30:14.559ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5748,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63162 req_id = 6b025f42-2bd8-44c7-8a05-7fd1487e4238
101072025-04-30T06:30:14.559ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
101082025-04-30T06:30:14.569ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5748, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63162 req_id = 6b025f42-2bd8-44c7-8a05-7fd1487e4238
101092025-04-30T06:30:14.569ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 91de8ef0-95ad-44df-9109-3223e1261929 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101102025-04-30T06:30:14.569ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 91de8ef0-95ad-44df-9109-3223e1261929 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101112025-04-30T06:30:14.571ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 91de8ef0-95ad-44df-9109-3223e1261929 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101122025-04-30T06:30:14.575ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
101132025-04-30T06:30:14.575ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
101142025-04-30T06:30:14.575ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
101152025-04-30T06:30:14.579ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101162025-04-30T06:30:14.586ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101172025-04-30T06:30:14.586ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101182025-04-30T06:30:14.586ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101192025-04-30T06:30:14.589ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a36f6fae-f5e8-4c9f-802b-a06bb4ff1d0a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101202025-04-30T06:30:14.589ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
101212025-04-30T06:30:14.589ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 30284 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 91de8ef0-95ad-44df-9109-3223e1261929 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101222025-04-30T06:30:14.589ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "91de8ef0-95ad-44df-9109-3223e1261929", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101232025-04-30T06:30:14.590ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
101242025-04-30T06:30:14.590ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 3f73f8e0-6b13-4438-9d7f-f408088a631c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
101252025-04-30T06:30:14.591ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 3f73f8e0-6b13-4438-9d7f-f408088a631c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
101262025-04-30T06:30:14.591ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 3f73f8e0-6b13-4438-9d7f-f408088a631c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
101272025-04-30T06:30:14.591ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 3f73f8e0-6b13-4438-9d7f-f408088a631c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
101282025-04-30T06:30:14.593ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec25c252-4d34-4f1c-9b24-c5fc11c9b201", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101292025-04-30T06:30:14.594ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
101302025-04-30T06:30:14.594ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 3838 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 3f73f8e0-6b13-4438-9d7f-f408088a631c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
101312025-04-30T06:30:14.599ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f73f8e0-6b13-4438-9d7f-f408088a631c", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101322025-04-30T06:30:14.599ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101332025-04-30T06:30:14.599ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
101342025-04-30T06:30:14.599ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
101352025-04-30T06:30:14.601ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60877c9b-2446-46ad-8da1-9a327c41607e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101362025-04-30T06:30:14.601ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
101372025-04-30T06:30:14.604ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdd2e2eb-4759-4025-b659-ec7306f5d0d3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101382025-04-30T06:30:14.604ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
101392025-04-30T06:30:14.608ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "161f7083-4ddc-4e5d-a2ed-01da25d27459", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101402025-04-30T06:30:14.608ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
101412025-04-30T06:30:14.610ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fba934a7-032d-4c95-af42-8e6d46e8c956", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101422025-04-30T06:30:14.610ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
101432025-04-30T06:30:14.618ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "629e4524-5a40-410a-955c-4bc9b17522d8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101442025-04-30T06:30:14.618ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
101452025-04-30T06:30:14.641ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f311e883-a17a-4789-b4b3-d0b44acd5680", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101462025-04-30T06:30:14.641ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
101472025-04-30T06:30:14.643ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35730781-ffa8-4230-8fc0-5845f9853240", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101482025-04-30T06:30:14.643ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
101492025-04-30T06:30:14.644ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a482ff0-2ed0-47f8-921e-628ef8d99c8f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101502025-04-30T06:30:14.644ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
101512025-04-30T06:30:14.645ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b73fcd1-8427-48bd-a179-4b34a8d92f61", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101522025-04-30T06:30:14.645ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
101532025-04-30T06:30:14.650ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48420f99-ad4c-4985-85c4-a2d6d64c1b1f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101542025-04-30T06:30:14.651ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
101552025-04-30T06:30:14.653ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d9bfd75-26ff-4422-99a8-51ff0148f2fc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101562025-04-30T06:30:14.654ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
101572025-04-30T06:30:14.660ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0949877-fadd-43dc-bb7c-713ffd96771e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101582025-04-30T06:30:14.661ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
101592025-04-30T06:30:14.667ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
101602025-04-30T06:30:14.670ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5beaddd-95cf-41f7-b7a5-ff15bff845f2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101612025-04-30T06:30:14.671ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
101622025-04-30T06:30:14.674ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "768a5c36-4456-443d-b4fc-92e305ec45f2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101632025-04-30T06:30:14.674ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
101642025-04-30T06:30:14.676ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "157c10b5-2c3a-46f1-9ec0-4de821c908ba", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101652025-04-30T06:30:14.676ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
101662025-04-30T06:30:14.679ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31ced6ca-9ece-412f-9938-3ebb2a690cdb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101672025-04-30T06:30:14.679ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
101682025-04-30T06:30:14.681ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "164dc91c-fcda-4a7c-9a98-1b00f849018a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101692025-04-30T06:30:14.681ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
101702025-04-30T06:30:14.682ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "581021ab-6762-416b-ab04-46902ccc06c3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101712025-04-30T06:30:14.682ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
101722025-04-30T06:30:14.684ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdcedfc9-17a7-41b6-a6d0-659005489346", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101732025-04-30T06:30:14.684ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
101742025-04-30T06:30:14.687ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e7688e9-31df-4e68-9d89-929699a2ac25", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101752025-04-30T06:30:14.687ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
101762025-04-30T06:30:14.688ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "232a9f1f-5909-45f8-92ab-82c8e8f0fe76", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101772025-04-30T06:30:14.688ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
101782025-04-30T06:30:14.689ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7da2fbfa-58b6-420f-91db-6df95a0f98d0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101792025-04-30T06:30:14.690ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
101802025-04-30T06:30:14.691ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c817ac1d-600f-489c-97e1-c3bc93b2b1a8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101812025-04-30T06:30:14.692ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
101822025-04-30T06:30:14.694ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d6b0fd7-272c-43bd-98ea-77782d81c2b5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101832025-04-30T06:30:14.694ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
101842025-04-30T06:30:14.696ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05be3abe-3a5c-4cfa-b81e-db9b59ab0d85", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101852025-04-30T06:30:14.696ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
101862025-04-30T06:30:14.701ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb54b57d-eaa4-4b4b-887d-55cab25b06d8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101872025-04-30T06:30:14.701ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
101882025-04-30T06:30:14.704ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "784d3481-ee7e-45a9-843a-94b41a50c0eb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101892025-04-30T06:30:14.704ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
101902025-04-30T06:30:14.707ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aeea93b8-3b59-457f-a210-34e5abe3f3be", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101912025-04-30T06:30:14.707ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
101922025-04-30T06:30:14.710ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a355bd1-ea56-4b8d-a50f-fd0fe57f7ee4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101932025-04-30T06:30:14.711ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
101942025-04-30T06:30:14.712ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80a0bb46-b620-4cb0-ac27-3d4cb40d083a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101952025-04-30T06:30:14.712ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
101962025-04-30T06:30:14.713ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cba392a0-66ab-4a1b-babc-9a760fa5ef84", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101972025-04-30T06:30:14.713ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
101982025-04-30T06:30:14.714ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaa4ee15-810e-4a05-ba77-d378841cb40d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
101992025-04-30T06:30:14.714ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
102002025-04-30T06:30:14.715ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65548bc4-4e04-4706-8bba-a4c3fd23fda6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
102012025-04-30T06:30:14.716ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
102022025-04-30T06:30:14.719ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102032025-04-30T06:30:14.719ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102042025-04-30T06:30:14.719ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102052025-04-30T06:30:14.719ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
102062025-04-30T06:30:14.720ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102072025-04-30T06:30:14.721ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f8a51a7-119c-427a-9665-ab30b04ba225", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
102082025-04-30T06:30:14.721ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
102092025-04-30T06:30:14.723ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102102025-04-30T06:30:14.728ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102112025-04-30T06:30:14.728ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102122025-04-30T06:30:14.728ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
102132025-04-30T06:30:14.728ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102142025-04-30T06:30:14.731ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10b1ead7-daf0-4f5d-93f9-7f4253944454", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
102152025-04-30T06:30:14.731ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
102162025-04-30T06:30:14.731ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102172025-04-30T06:30:14.731ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
102182025-04-30T06:30:14.733ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102192025-04-30T06:30:14.736ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43444dd4-100d-4c63-b864-09f3b599a666", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
102202025-04-30T06:30:14.737ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
102212025-04-30T06:30:14.740ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "44893a4d-dab3-4010-acad-3ce0c9422335", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
102222025-04-30T06:30:14.740ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
102232025-04-30T06:30:14.745ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7373412-c24f-4f3f-817c-3c27ac486e5b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
102242025-04-30T06:30:14.746ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
102252025-04-30T06:30:14.750ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
102262025-04-30T06:30:14.755ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
102272025-04-30T06:30:14.756ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
102282025-04-30T06:30:14.758ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
102292025-04-30T06:30:14.762ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "033383f1-96e3-4fb5-a3a1-d3c8aba41591", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
102302025-04-30T06:30:14.762ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
102312025-04-30T06:30:14.766ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
102322025-04-30T06:30:14.766ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
102332025-04-30T06:30:14.766ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cecc31e0-e38d-4f54-9e4a-7623a806ffd4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
102342025-04-30T06:30:14.766ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
102352025-04-30T06:30:14.769ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff25dd59-e6bb-4b1b-b0de-fee926e47e8b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
102362025-04-30T06:30:14.769ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
102372025-04-30T06:30:14.775ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
102382025-04-30T06:30:14.775ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
102392025-04-30T06:30:14.776ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e38a0b2a-36a1-449e-8377-6c7774225535", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
102402025-04-30T06:30:14.776ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
102412025-04-30T06:30:14.777ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3336b8fd-444d-4104-b4af-9a0963c5219f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
102422025-04-30T06:30:14.777ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
102432025-04-30T06:30:14.777ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102442025-04-30T06:30:14.777ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102452025-04-30T06:30:14.778ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102462025-04-30T06:30:14.780ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102472025-04-30T06:30:14.785ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102482025-04-30T06:30:14.803ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
102492025-04-30T06:30:14.803ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
102502025-04-30T06:30:14.803ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102512025-04-30T06:30:14.803ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102522025-04-30T06:30:14.803ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102532025-04-30T06:30:14.804ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102542025-04-30T06:30:14.815ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102552025-04-30T06:30:14.819ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
102562025-04-30T06:30:14.819ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102572025-04-30T06:30:14.819ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102582025-04-30T06:30:14.820ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102592025-04-30T06:30:14.821ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
102602025-04-30T06:30:14.895ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
102612025-04-30T06:30:14.895ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
102622025-04-30T06:30:14.896ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102632025-04-30T06:30:14.896ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102642025-04-30T06:30:14.896ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102652025-04-30T06:30:14.896ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
102662025-04-30T06:30:14.898ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102672025-04-30T06:30:14.898ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:38033 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
102682025-04-30T06:30:14.900ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
102692025-04-30T06:30:14.901ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:38033 req_id = 47be48a3-4aba-4f9f-a6e7-7c8a43f9423c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
102702025-04-30T06:30:14.903ZINFOtest_image_promotion (dropshot): request completed latency_us = 1634 local_addr = [::1]:60070 method = GET remote_addr = [::1]:38033 req_id = 47be48a3-4aba-4f9f-a6e7-7c8a43f9423c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
102712025-04-30T06:30:14.903ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102722025-04-30T06:30:14.915ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102732025-04-30T06:30:14.915ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102742025-04-30T06:30:14.915ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
102752025-04-30T06:30:14.915ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102762025-04-30T06:30:14.916ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "47be48a3-4aba-4f9f-a6e7-7c8a43f9423c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
102772025-04-30T06:30:14.918ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
102782025-04-30T06:30:14.918ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
102792025-04-30T06:30:14.918ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
102802025-04-30T06:30:14.922ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
102812025-04-30T06:30:14.923ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102822025-04-30T06:30:14.923ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102832025-04-30T06:30:14.923ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102842025-04-30T06:30:14.924ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102852025-04-30T06:30:14.927ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102862025-04-30T06:30:14.927ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
102872025-04-30T06:30:14.928ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102882025-04-30T06:30:14.931ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102892025-04-30T06:30:14.937ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
102902025-04-30T06:30:14.937ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102912025-04-30T06:30:14.937ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102922025-04-30T06:30:14.938ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102932025-04-30T06:30:15.015ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
102942025-04-30T06:30:15.016ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
102952025-04-30T06:30:15.016ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 683.33462ms iteration = 36 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
102962025-04-30T06:30:15.016ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 37 reason = Signaled
102972025-04-30T06:30:15.016ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
102982025-04-30T06:30:15.016ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102992025-04-30T06:30:15.016ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103002025-04-30T06:30:15.016ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103012025-04-30T06:30:15.016ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103022025-04-30T06:30:15.021ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103032025-04-30T06:30:15.025ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
103042025-04-30T06:30:15.025ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103052025-04-30T06:30:15.025ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103062025-04-30T06:30:15.026ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103072025-04-30T06:30:15.029ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
103082025-04-30T06:30:15.031ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
103092025-04-30T06:30:15.031ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
103102025-04-30T06:30:15.032ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 34 a8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 4............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50072 req_id = 61979ed4-11b6-4d0d-a534-034596e72e3e
103112025-04-30T06:30:15.033ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13480,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50072 req_id = 61979ed4-11b6-4d0d-a534-034596e72e3e
103122025-04-30T06:30:15.033ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
103132025-04-30T06:30:15.033ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103142025-04-30T06:30:15.033ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50072 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 61979ed4-11b6-4d0d-a534-034596e72e3e
103152025-04-30T06:30:15.033ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3479, tv_nsec: 832859762 } })
103162025-04-30T06:30:15.034ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
103172025-04-30T06:30:15.034ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6d e6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e m...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61372 req_id = 135a978e-77c4-4c11-820f-e81abf01fdcc
103182025-04-30T06:30:15.034ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28134,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61372 req_id = 135a978e-77c4-4c11-820f-e81abf01fdcc
103192025-04-30T06:30:15.034ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103202025-04-30T06:30:15.034ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61372 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 135a978e-77c4-4c11-820f-e81abf01fdcc
103212025-04-30T06:30:15.109ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
103222025-04-30T06:30:15.109ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103232025-04-30T06:30:15.109ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103242025-04-30T06:30:15.109ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103252025-04-30T06:30:15.109ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
103262025-04-30T06:30:15.111ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103272025-04-30T06:30:15.112ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2b eb 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c +............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62261 req_id = 1910c8d6-d805-4f7c-ae80-506369959424
103282025-04-30T06:30:15.112ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11243,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62261 req_id = 1910c8d6-d805-4f7c-ae80-506369959424
103292025-04-30T06:30:15.112ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
103302025-04-30T06:30:15.112ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11243, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62261 req_id = 1910c8d6-d805-4f7c-ae80-506369959424
103312025-04-30T06:30:15.113ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
103322025-04-30T06:30:15.113ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
103332025-04-30T06:30:15.113ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
103342025-04-30T06:30:15.115ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103352025-04-30T06:30:15.125ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
103362025-04-30T06:30:15.125ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103372025-04-30T06:30:15.125ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
103382025-04-30T06:30:15.126ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103392025-04-30T06:30:15.131ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103402025-04-30T06:30:15.131ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
103412025-04-30T06:30:15.137ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103422025-04-30T06:30:15.137ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9de6d293-9aa8-4ceb-90f6-c778990ce0e1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103432025-04-30T06:30:15.138ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
103442025-04-30T06:30:15.142ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1dc1428-49ab-4ef0-b396-aea88bd815a8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103452025-04-30T06:30:15.142ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
103462025-04-30T06:30:15.147ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec57a21a-d34c-4879-9deb-590135fd89b7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103472025-04-30T06:30:15.147ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
103482025-04-30T06:30:15.150ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eeee945a-21b5-4445-befc-035c408ba254", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103492025-04-30T06:30:15.150ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
103502025-04-30T06:30:15.165ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "23158751-37a8-4db9-9224-d4032175b0b3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103512025-04-30T06:30:15.166ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
103522025-04-30T06:30:15.172ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50bad913-cd13-4dc6-91b8-8a139d9fa9e2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103532025-04-30T06:30:15.172ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
103542025-04-30T06:30:15.189ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e91d95e8-87a5-4b4b-86bd-c64e70222c1a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103552025-04-30T06:30:15.190ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
103562025-04-30T06:30:15.192ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48411fee-bc7a-4ec3-8a9a-812e82c6c56f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103572025-04-30T06:30:15.192ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
103582025-04-30T06:30:15.193ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03792e8c-4973-4db7-a7ff-06395d4753c8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103592025-04-30T06:30:15.193ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
103602025-04-30T06:30:15.197ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "390bc093-fcfd-4845-bab6-aa5e551bdad1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103612025-04-30T06:30:15.197ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
103622025-04-30T06:30:15.199ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28e85726-c734-4695-be46-8468b72e61a2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103632025-04-30T06:30:15.199ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
103642025-04-30T06:30:15.200ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad8d30b1-1147-42c9-bd20-a9f7158aa760", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103652025-04-30T06:30:15.201ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
103662025-04-30T06:30:15.202ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cacad5ef-283d-47fb-ba56-a7034d246fdd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103672025-04-30T06:30:15.202ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
103682025-04-30T06:30:15.205ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e6fcccb-dc24-4882-a93d-2ce7d1057603", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103692025-04-30T06:30:15.206ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
103702025-04-30T06:30:15.211ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "086d854f-e6d2-4849-9a35-3fb74fdef164", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103712025-04-30T06:30:15.211ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
103722025-04-30T06:30:15.214ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7821bc4-7b14-47c6-910e-38c4731e0b1a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103732025-04-30T06:30:15.214ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
103742025-04-30T06:30:15.216ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ab8cfb9-25dd-436d-8492-1a2c3a566916", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103752025-04-30T06:30:15.216ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
103762025-04-30T06:30:15.219ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdb2e40a-e0f0-4d9f-bd7d-456815fdd99b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103772025-04-30T06:30:15.219ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
103782025-04-30T06:30:15.223ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
103792025-04-30T06:30:15.224ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a36bd77-d0f9-40a3-81a0-2905beb7f89a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103802025-04-30T06:30:15.224ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
103812025-04-30T06:30:15.225ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07b437d2-9bfd-4db4-9d57-d48d34f615f8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103822025-04-30T06:30:15.225ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
103832025-04-30T06:30:15.228ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a76407fc-9638-465d-9f55-044830bdaa8f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103842025-04-30T06:30:15.228ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
103852025-04-30T06:30:15.230ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01455c65-071c-4920-9873-152a25956280", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103862025-04-30T06:30:15.230ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
103872025-04-30T06:30:15.231ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98cef596-328f-46f5-a2f6-94a44a7afdb9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103882025-04-30T06:30:15.231ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
103892025-04-30T06:30:15.233ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33830518-d130-46f7-8d10-72c086ed4346", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103902025-04-30T06:30:15.233ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
103912025-04-30T06:30:15.235ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf8af27e-8529-40d4-9915-7292003973c6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103922025-04-30T06:30:15.235ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
103932025-04-30T06:30:15.237ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0461779-134d-49ae-b0e2-725733fe9a54", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103942025-04-30T06:30:15.237ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
103952025-04-30T06:30:15.240ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "962f5f75-f30b-4e2d-af62-4819897a0ac2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103962025-04-30T06:30:15.240ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
103972025-04-30T06:30:15.243ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "958078bd-3ffa-48a1-96e1-f10c5c2be43e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
103982025-04-30T06:30:15.243ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
103992025-04-30T06:30:15.245ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bb75ce6-7715-47db-b8b0-dcb979705e62", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104002025-04-30T06:30:15.245ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
104012025-04-30T06:30:15.247ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "15dcd42b-e67a-404f-9ec8-6817d16e8577", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104022025-04-30T06:30:15.247ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
104032025-04-30T06:30:15.248ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41a3207a-b710-4fbf-a31c-e67b1c0b1344", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104042025-04-30T06:30:15.248ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
104052025-04-30T06:30:15.249ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0b1660b-11b3-4158-9482-6e4445c32b0e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104062025-04-30T06:30:15.249ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
104072025-04-30T06:30:15.250ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf6f296e-8b1a-46d6-87b1-6155aae56363", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104082025-04-30T06:30:15.250ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
104092025-04-30T06:30:15.251ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18a78f6e-afef-4122-b16a-dba7ee45d761", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104102025-04-30T06:30:15.251ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
104112025-04-30T06:30:15.253ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c835f40c-5eea-43fb-97f2-e2860512c4f5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104122025-04-30T06:30:15.253ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
104132025-04-30T06:30:15.255ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17455c6f-8301-48d2-9b65-b673fecc780e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104142025-04-30T06:30:15.255ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
104152025-04-30T06:30:15.257ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6607f41f-f967-4e1f-ada8-677cbb0a0a34", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104162025-04-30T06:30:15.257ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
104172025-04-30T06:30:15.259ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68be1144-4cc7-4b07-9972-ba9b03bac23c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104182025-04-30T06:30:15.259ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
104192025-04-30T06:30:15.261ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0156d77-203d-4887-96f8-86394be2205e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104202025-04-30T06:30:15.261ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
104212025-04-30T06:30:15.266ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6c88b24-8537-49b3-9b66-9ec8dd69fb95", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104222025-04-30T06:30:15.266ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
104232025-04-30T06:30:15.268ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fab42d3c-94e0-416a-b847-f58ff367b5d1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104242025-04-30T06:30:15.268ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
104252025-04-30T06:30:15.269ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6e840f0-5b33-447c-8daf-e8247fd68651", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104262025-04-30T06:30:15.269ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
104272025-04-30T06:30:15.270ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c87176b-cc7e-4f7a-8ce8-3fb6f2f8dcac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104282025-04-30T06:30:15.270ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
104292025-04-30T06:30:15.271ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d38979c2-46a3-49f9-a4c2-be24467989b7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104302025-04-30T06:30:15.271ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
104312025-04-30T06:30:15.272ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f78021ea-bda6-46b6-bb3b-316783e1aae6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104322025-04-30T06:30:15.272ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
104332025-04-30T06:30:15.272ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104342025-04-30T06:30:15.272ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104352025-04-30T06:30:15.272ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104362025-04-30T06:30:15.273ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104372025-04-30T06:30:15.274ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104382025-04-30T06:30:15.274ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104392025-04-30T06:30:15.274ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104402025-04-30T06:30:15.274ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
104412025-04-30T06:30:15.276ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104422025-04-30T06:30:15.277ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104432025-04-30T06:30:15.282ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
104442025-04-30T06:30:15.282ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
104452025-04-30T06:30:15.282ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104462025-04-30T06:30:15.282ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104472025-04-30T06:30:15.282ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104482025-04-30T06:30:15.282ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104492025-04-30T06:30:15.288ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104502025-04-30T06:30:15.297ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
104512025-04-30T06:30:15.297ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104522025-04-30T06:30:15.297ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104532025-04-30T06:30:15.297ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104542025-04-30T06:30:15.298ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104552025-04-30T06:30:15.304ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104562025-04-30T06:30:15.304ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104572025-04-30T06:30:15.304ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
104582025-04-30T06:30:15.305ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104592025-04-30T06:30:15.364ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
104602025-04-30T06:30:15.364ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
104612025-04-30T06:30:15.365ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:43154 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
104622025-04-30T06:30:15.365ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:43154 req_id = 81315ac7-599e-42d7-a3ff-432a893b3978 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
104632025-04-30T06:30:15.365ZINFOtest_image_promotion (dropshot): request completed latency_us = 407 local_addr = [::1]:60070 method = GET remote_addr = [::1]:43154 req_id = 81315ac7-599e-42d7-a3ff-432a893b3978 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
104642025-04-30T06:30:15.366ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "81315ac7-599e-42d7-a3ff-432a893b3978", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:14 GMT"} })
104652025-04-30T06:30:15.369ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
104662025-04-30T06:30:15.369ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
104672025-04-30T06:30:15.369ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
104682025-04-30T06:30:15.372ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
104692025-04-30T06:30:15.372ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104702025-04-30T06:30:15.372ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104712025-04-30T06:30:15.372ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104722025-04-30T06:30:15.373ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104732025-04-30T06:30:15.373ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104742025-04-30T06:30:15.373ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
104752025-04-30T06:30:15.375ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104762025-04-30T06:30:15.390ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104772025-04-30T06:30:15.394ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
104782025-04-30T06:30:15.395ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104792025-04-30T06:30:15.395ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104802025-04-30T06:30:15.396ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104812025-04-30T06:30:15.463ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
104822025-04-30T06:30:15.483ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
104832025-04-30T06:30:15.483ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
104842025-04-30T06:30:15.484ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 468.494575ms iteration = 37 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
104852025-04-30T06:30:15.486ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 38 reason = Signaled
104862025-04-30T06:30:15.486ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
104872025-04-30T06:30:15.486ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104882025-04-30T06:30:15.486ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104892025-04-30T06:30:15.486ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104902025-04-30T06:30:15.487ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104912025-04-30T06:30:15.491ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104922025-04-30T06:30:15.496ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
104932025-04-30T06:30:15.496ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104942025-04-30T06:30:15.496ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104952025-04-30T06:30:15.497ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104962025-04-30T06:30:15.503ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
104972025-04-30T06:30:15.503ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
104982025-04-30T06:30:15.504ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fe 9a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49406 req_id = 578c4703-ba5e-4786-b483-acb1810c9395
104992025-04-30T06:30:15.504ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65178,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49406 req_id = 578c4703-ba5e-4786-b483-acb1810c9395
105002025-04-30T06:30:15.505ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
105012025-04-30T06:30:15.505ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
105022025-04-30T06:30:15.505ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49406 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 578c4703-ba5e-4786-b483-acb1810c9395
105032025-04-30T06:30:15.505ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3480, tv_nsec: 304886705 } })
105042025-04-30T06:30:15.506ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
105052025-04-30T06:30:15.506ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d7 68 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .h..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57217 req_id = 0ab3ebb0-6db7-4c46-9c98-e07198716e52
105062025-04-30T06:30:15.506ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55144,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57217 req_id = 0ab3ebb0-6db7-4c46-9c98-e07198716e52
105072025-04-30T06:30:15.506ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
105082025-04-30T06:30:15.506ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57217 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0ab3ebb0-6db7-4c46-9c98-e07198716e52
105092025-04-30T06:30:15.594ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
105102025-04-30T06:30:15.594ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105112025-04-30T06:30:15.594ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105122025-04-30T06:30:15.594ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105132025-04-30T06:30:15.595ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
105142025-04-30T06:30:15.598ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105152025-04-30T06:30:15.598ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105162025-04-30T06:30:15.598ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105172025-04-30T06:30:15.599ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 0b 34 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .4..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59033 req_id = a092b28d-9336-4d65-b8e1-8886e3117a6a
105182025-04-30T06:30:15.600ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2868,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59033 req_id = a092b28d-9336-4d65-b8e1-8886e3117a6a
105192025-04-30T06:30:15.600ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
105202025-04-30T06:30:15.600ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2868, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59033 req_id = a092b28d-9336-4d65-b8e1-8886e3117a6a
105212025-04-30T06:30:15.602ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
105222025-04-30T06:30:15.602ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
105232025-04-30T06:30:15.602ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
105242025-04-30T06:30:15.603ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105252025-04-30T06:30:15.603ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105262025-04-30T06:30:15.604ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105272025-04-30T06:30:15.604ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105282025-04-30T06:30:15.604ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105292025-04-30T06:30:15.611ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105302025-04-30T06:30:15.611ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105312025-04-30T06:30:15.611ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
105322025-04-30T06:30:15.611ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105332025-04-30T06:30:15.614ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105342025-04-30T06:30:15.614ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
105352025-04-30T06:30:15.616ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105362025-04-30T06:30:15.616ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ba21c65-c63c-4527-a680-c50ae67d13fd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105372025-04-30T06:30:15.617ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
105382025-04-30T06:30:15.619ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9330e0e3-9f3f-4aa2-9027-32a808ff580f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105392025-04-30T06:30:15.620ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
105402025-04-30T06:30:15.624ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "282d9011-af89-4196-80d4-26c96212ac51", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105412025-04-30T06:30:15.624ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
105422025-04-30T06:30:15.628ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6523a4d-380a-4a12-a4ad-d53cc49d025b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105432025-04-30T06:30:15.629ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
105442025-04-30T06:30:15.633ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6750dd3-10e0-4c36-9535-6043b9537079", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105452025-04-30T06:30:15.634ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
105462025-04-30T06:30:15.641ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0e5ff62-4d35-40b6-bde3-4332844a7e23", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105472025-04-30T06:30:15.641ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
105482025-04-30T06:30:15.648ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "129d5cca-32a1-4472-bd0f-d3eef1c257ea", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105492025-04-30T06:30:15.649ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
105502025-04-30T06:30:15.650ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
105512025-04-30T06:30:15.652ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5af9fd60-cf02-48d9-8915-c06a56d348b2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105522025-04-30T06:30:15.652ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
105532025-04-30T06:30:15.653ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db14167a-59a3-44f4-8c3b-cb6c0f60cabb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105542025-04-30T06:30:15.653ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
105552025-04-30T06:30:15.654ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e67d4efe-f250-46a3-a63e-b2c4f996e706", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105562025-04-30T06:30:15.655ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
105572025-04-30T06:30:15.656ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "691898a0-a78d-4550-9046-790158dba157", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105582025-04-30T06:30:15.656ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
105592025-04-30T06:30:15.657ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dcf1eb3-014c-4484-8807-732500128d32", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105602025-04-30T06:30:15.657ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
105612025-04-30T06:30:15.658ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b97b8e5-8e39-44bf-8be0-70bcca8401be", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105622025-04-30T06:30:15.658ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
105632025-04-30T06:30:15.660ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd4b82e3-669f-4fce-a891-4cec677b1d6b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105642025-04-30T06:30:15.660ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
105652025-04-30T06:30:15.662ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b79a121-01e0-4403-a166-3650534c1ccc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105662025-04-30T06:30:15.662ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
105672025-04-30T06:30:15.664ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "35f95b2c-28b0-4c55-9700-b4c855f2da3d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105682025-04-30T06:30:15.665ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
105692025-04-30T06:30:15.668ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66eaa21e-db23-4f6c-b1da-831a20fe4c59", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105702025-04-30T06:30:15.668ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
105712025-04-30T06:30:15.672ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4c021ff-8012-4b88-b818-373e63f5ea44", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105722025-04-30T06:30:15.672ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
105732025-04-30T06:30:15.674ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95c0be6b-f9b4-44b8-93a9-98d539004d06", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105742025-04-30T06:30:15.674ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
105752025-04-30T06:30:15.679ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c260eae2-fd8d-4abf-8b25-49e696e0a932", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105762025-04-30T06:30:15.680ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
105772025-04-30T06:30:15.681ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e14922d-00bc-456d-8f25-9bd3d33abc9b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105782025-04-30T06:30:15.681ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
105792025-04-30T06:30:15.682ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b48ad0f9-391f-4875-93c3-a4709513eed6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105802025-04-30T06:30:15.682ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
105812025-04-30T06:30:15.683ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bf1e207-9176-4377-a012-c2be02fc9f13", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105822025-04-30T06:30:15.683ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
105832025-04-30T06:30:15.685ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f98556f8-6e9e-421a-8643-c8ee4688cbb9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105842025-04-30T06:30:15.685ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
105852025-04-30T06:30:15.687ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c904b07-354f-47ef-9ecc-e43b6579c33a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105862025-04-30T06:30:15.687ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
105872025-04-30T06:30:15.689ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ba28957-d01b-424f-9472-c6893fd0a129", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105882025-04-30T06:30:15.689ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
105892025-04-30T06:30:15.691ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9062e50-47b1-4732-9f99-7d6c2e53ef17", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105902025-04-30T06:30:15.691ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
105912025-04-30T06:30:15.697ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d920333-02f9-40f0-b04f-b73fb4f9b39b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
105922025-04-30T06:30:15.697ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
105932025-04-30T06:30:15.700ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105942025-04-30T06:30:15.701ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105952025-04-30T06:30:15.701ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105962025-04-30T06:30:15.701ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
105972025-04-30T06:30:15.702ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105982025-04-30T06:30:15.708ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105992025-04-30T06:30:15.716ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106002025-04-30T06:30:15.716ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106012025-04-30T06:30:15.716ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
106022025-04-30T06:30:15.717ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106032025-04-30T06:30:15.717ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d00dead-dd20-4b8c-9895-557b7a1065bc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106042025-04-30T06:30:15.718ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
106052025-04-30T06:30:15.719ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106062025-04-30T06:30:15.719ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
106072025-04-30T06:30:15.728ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106082025-04-30T06:30:15.734ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "272a3d8e-d828-4c4b-9279-881a2e56465b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106092025-04-30T06:30:15.734ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
106102025-04-30T06:30:15.736ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5024bc9-4750-4902-833e-03e5de409cd0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106112025-04-30T06:30:15.736ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
106122025-04-30T06:30:15.738ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2c303ed-2cc6-49c8-ae5e-2394660fd476", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106132025-04-30T06:30:15.739ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
106142025-04-30T06:30:15.740ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a7a09a2-2549-4a32-a73d-310a373ccf36", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106152025-04-30T06:30:15.740ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
106162025-04-30T06:30:15.741ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7de9dd07-011a-42cb-baa9-fa683270872f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106172025-04-30T06:30:15.758ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
106182025-04-30T06:30:15.759ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106192025-04-30T06:30:15.759ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106202025-04-30T06:30:15.760ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
106212025-04-30T06:30:15.761ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
106222025-04-30T06:30:15.763ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc31d3e5-5970-4fad-bde3-d3d7d108b2e3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106232025-04-30T06:30:15.763ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
106242025-04-30T06:30:15.766ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
106252025-04-30T06:30:15.766ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
106262025-04-30T06:30:15.770ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
106272025-04-30T06:30:15.770ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106282025-04-30T06:30:15.770ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ecacbfb-a490-47d5-9136-da5550138ab5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106292025-04-30T06:30:15.770ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
106302025-04-30T06:30:15.774ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9f6a60e-570c-4067-942f-7f6d81804689", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106312025-04-30T06:30:15.774ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
106322025-04-30T06:30:15.778ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab99695c-3630-4e0b-bb47-735c58caa2ce", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106332025-04-30T06:30:15.778ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
106342025-04-30T06:30:15.783ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
106352025-04-30T06:30:15.785ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ea7db13-479f-49ef-bd06-9528b0b8ba60", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106362025-04-30T06:30:15.785ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
106372025-04-30T06:30:15.788ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e3eaf6b-4ec8-49b9-a17f-19200e410ee7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106382025-04-30T06:30:15.789ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
106392025-04-30T06:30:15.791ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdc427cc-4bc5-4c89-8566-555fad247df4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106402025-04-30T06:30:15.791ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
106412025-04-30T06:30:15.793ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad74fc9a-b322-4940-bb79-1c3627a4cfc4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106422025-04-30T06:30:15.793ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
106432025-04-30T06:30:15.794ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a891e7b2-cec6-49d7-95de-be7789957a77", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106442025-04-30T06:30:15.794ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
106452025-04-30T06:30:15.796ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76dffed8-cf14-4ecc-8e61-f66461e69c64", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106462025-04-30T06:30:15.796ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
106472025-04-30T06:30:15.798ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "912bc3de-0367-4a6a-bb16-db7614febb99", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106482025-04-30T06:30:15.798ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
106492025-04-30T06:30:15.798ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106502025-04-30T06:30:15.798ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106512025-04-30T06:30:15.798ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106522025-04-30T06:30:15.800ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106532025-04-30T06:30:15.804ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106542025-04-30T06:30:15.810ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
106552025-04-30T06:30:15.813ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
106562025-04-30T06:30:15.813ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106572025-04-30T06:30:15.813ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106582025-04-30T06:30:15.813ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106592025-04-30T06:30:15.813ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106602025-04-30T06:30:15.816ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106612025-04-30T06:30:15.820ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
106622025-04-30T06:30:15.820ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106632025-04-30T06:30:15.820ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106642025-04-30T06:30:15.821ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106652025-04-30T06:30:15.923ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
106662025-04-30T06:30:15.923ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
106672025-04-30T06:30:15.924ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106682025-04-30T06:30:15.924ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106692025-04-30T06:30:15.924ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106702025-04-30T06:30:15.924ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
106712025-04-30T06:30:15.925ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106722025-04-30T06:30:15.925ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:45688 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
106732025-04-30T06:30:15.926ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
106742025-04-30T06:30:15.926ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:45688 req_id = 381cf419-a640-4928-bea6-885ef0f4c50a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
106752025-04-30T06:30:15.927ZINFOtest_image_promotion (dropshot): request completed latency_us = 531 local_addr = [::1]:60070 method = GET remote_addr = [::1]:45688 req_id = 381cf419-a640-4928-bea6-885ef0f4c50a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
106762025-04-30T06:30:15.927ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "381cf419-a640-4928-bea6-885ef0f4c50a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
106772025-04-30T06:30:15.931ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
106782025-04-30T06:30:15.931ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
106792025-04-30T06:30:15.931ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
106802025-04-30T06:30:15.935ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
106812025-04-30T06:30:15.935ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106822025-04-30T06:30:15.935ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106832025-04-30T06:30:15.935ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106842025-04-30T06:30:15.936ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106852025-04-30T06:30:15.937ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106862025-04-30T06:30:15.942ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106872025-04-30T06:30:15.942ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106882025-04-30T06:30:15.942ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
106892025-04-30T06:30:15.943ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106902025-04-30T06:30:15.948ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106912025-04-30T06:30:15.954ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
106922025-04-30T06:30:15.954ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106932025-04-30T06:30:15.954ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106942025-04-30T06:30:15.955ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106952025-04-30T06:30:15.956ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106962025-04-30T06:30:15.956ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
106972025-04-30T06:30:15.957ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106982025-04-30T06:30:16.014ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
106992025-04-30T06:30:16.015ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
107002025-04-30T06:30:16.015ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 529.188403ms iteration = 38 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
107012025-04-30T06:30:16.015ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 39 reason = Signaled
107022025-04-30T06:30:16.015ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
107032025-04-30T06:30:16.015ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107042025-04-30T06:30:16.015ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107052025-04-30T06:30:16.015ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107062025-04-30T06:30:16.016ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107072025-04-30T06:30:16.018ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
107082025-04-30T06:30:16.019ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107092025-04-30T06:30:16.025ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
107102025-04-30T06:30:16.026ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107112025-04-30T06:30:16.026ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107122025-04-30T06:30:16.026ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107132025-04-30T06:30:16.030ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
107142025-04-30T06:30:16.030ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
107152025-04-30T06:30:16.032ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c3 83 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54944 req_id = 7c7eab6d-679e-4302-9290-924bc8c0f5db
107162025-04-30T06:30:16.032ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50051,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54944 req_id = 7c7eab6d-679e-4302-9290-924bc8c0f5db
107172025-04-30T06:30:16.036ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
107182025-04-30T06:30:16.036ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
107192025-04-30T06:30:16.036ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54944 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7c7eab6d-679e-4302-9290-924bc8c0f5db
107202025-04-30T06:30:16.037ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3480, tv_nsec: 835974031 } })
107212025-04-30T06:30:16.037ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
107222025-04-30T06:30:16.037ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6e 86 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e n...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55211 req_id = 23543753-37a3-4d2f-9b06-5610bb9c689b
107232025-04-30T06:30:16.037ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28294,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55211 req_id = 23543753-37a3-4d2f-9b06-5610bb9c689b
107242025-04-30T06:30:16.037ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
107252025-04-30T06:30:16.037ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55211 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 23543753-37a3-4d2f-9b06-5610bb9c689b
107262025-04-30T06:30:16.096ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
107272025-04-30T06:30:16.097ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
107282025-04-30T06:30:16.097ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107292025-04-30T06:30:16.097ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107302025-04-30T06:30:16.097ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
107312025-04-30T06:30:16.098ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107322025-04-30T06:30:16.098ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 96 b5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64507 req_id = 2320c700-dfe6-43fa-bbe8-544ac15636b2
107332025-04-30T06:30:16.098ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38581,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64507 req_id = 2320c700-dfe6-43fa-bbe8-544ac15636b2
107342025-04-30T06:30:16.099ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
107352025-04-30T06:30:16.099ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38581, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64507 req_id = 2320c700-dfe6-43fa-bbe8-544ac15636b2
107362025-04-30T06:30:16.099ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
107372025-04-30T06:30:16.099ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
107382025-04-30T06:30:16.099ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
107392025-04-30T06:30:16.101ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107402025-04-30T06:30:16.107ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107412025-04-30T06:30:16.107ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107422025-04-30T06:30:16.107ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
107432025-04-30T06:30:16.107ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107442025-04-30T06:30:16.111ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5771b61-e999-4583-9056-5caf2bcb607a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107452025-04-30T06:30:16.111ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
107462025-04-30T06:30:16.112ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "502c198b-bd23-4e73-9f60-85638a282bcd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107472025-04-30T06:30:16.112ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
107482025-04-30T06:30:16.113ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107492025-04-30T06:30:16.113ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
107502025-04-30T06:30:16.113ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107512025-04-30T06:30:16.115ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea948fe0-449c-4042-9eab-6c9cbda666a3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107522025-04-30T06:30:16.116ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
107532025-04-30T06:30:16.118ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "510989ee-29b5-4a56-b834-e53598742636", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107542025-04-30T06:30:16.118ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
107552025-04-30T06:30:16.124ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fb9dd27-5b2c-4bc9-932b-59324c8a5e79", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107562025-04-30T06:30:16.124ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
107572025-04-30T06:30:16.126ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "33393bbb-584d-4698-bb0a-b3a9e6abbe6d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107582025-04-30T06:30:16.126ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
107592025-04-30T06:30:16.129ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5c2bb6a-6552-4298-b712-580ba4faaf9b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107602025-04-30T06:30:16.129ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
107612025-04-30T06:30:16.131ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c6acc9e-6273-4c30-925d-72d3e7644394", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107622025-04-30T06:30:16.131ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
107632025-04-30T06:30:16.132ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9662a813-addb-4dbc-a95a-233f7e447b97", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107642025-04-30T06:30:16.133ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
107652025-04-30T06:30:16.134ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c47daae-90dd-476e-a780-b7286a1e4908", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107662025-04-30T06:30:16.134ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
107672025-04-30T06:30:16.137ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ece2f3b-1cfc-4337-ae38-4bd84700d15f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107682025-04-30T06:30:16.137ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
107692025-04-30T06:30:16.139ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e2ab1b9-6266-441d-8b49-8159c5f1c905", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107702025-04-30T06:30:16.139ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
107712025-04-30T06:30:16.140ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b6ab110-405b-4bbb-b033-0c7de190902d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107722025-04-30T06:30:16.140ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
107732025-04-30T06:30:16.144ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
107742025-04-30T06:30:16.146ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66c7c1b2-911c-4162-87b9-4309340d3261", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107752025-04-30T06:30:16.146ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
107762025-04-30T06:30:16.148ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b91f954-c8b5-43db-9aa5-7403f76f72ae", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107772025-04-30T06:30:16.148ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
107782025-04-30T06:30:16.152ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c646ef25-3279-45c5-89f6-2cfa2e859753", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107792025-04-30T06:30:16.152ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
107802025-04-30T06:30:16.155ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86579dca-9119-42a1-9b1a-70077b61ee18", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107812025-04-30T06:30:16.155ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
107822025-04-30T06:30:16.158ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bab670ce-e61e-4a0b-b4cf-fdc7ae9b7f16", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107832025-04-30T06:30:16.159ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
107842025-04-30T06:30:16.161ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "360327cc-1138-4fe5-9059-17901318d8a8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107852025-04-30T06:30:16.161ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
107862025-04-30T06:30:16.162ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "176b64c2-57c2-4862-aed1-4f59d356ba5a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107872025-04-30T06:30:16.162ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
107882025-04-30T06:30:16.163ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8070d659-03dc-46a0-af70-c02ef452ed51", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107892025-04-30T06:30:16.163ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
107902025-04-30T06:30:16.166ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f9817ac-8e6f-4a8f-8832-e8856f844f30", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107912025-04-30T06:30:16.166ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
107922025-04-30T06:30:16.170ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a200b53-e673-477f-b19b-3767261a324d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107932025-04-30T06:30:16.170ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
107942025-04-30T06:30:16.172ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1524bd45-e0c1-4160-bf8f-400a80b1cf69", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107952025-04-30T06:30:16.172ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
107962025-04-30T06:30:16.183ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a19c388-2d00-4054-ae3e-c8452b97bd37", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107972025-04-30T06:30:16.183ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
107982025-04-30T06:30:16.188ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f23740d-fe69-4030-89af-546d26aa3d5f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
107992025-04-30T06:30:16.188ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
108002025-04-30T06:30:16.193ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbe087bf-64a2-4d44-b43f-3edde022d9b0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108012025-04-30T06:30:16.193ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
108022025-04-30T06:30:16.195ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
108032025-04-30T06:30:16.195ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108042025-04-30T06:30:16.195ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108052025-04-30T06:30:16.195ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
108062025-04-30T06:30:16.196ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108072025-04-30T06:30:16.199ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108082025-04-30T06:30:16.219ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
108092025-04-30T06:30:16.219ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108102025-04-30T06:30:16.219ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
108112025-04-30T06:30:16.220ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108122025-04-30T06:30:16.221ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffc0b6c0-b997-4bf7-a8dd-bfdfd50c65e6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108132025-04-30T06:30:16.221ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
108142025-04-30T06:30:16.224ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108152025-04-30T06:30:16.224ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
108162025-04-30T06:30:16.226ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108172025-04-30T06:30:16.229ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdc77983-b262-41c1-a894-93bb250d5fcc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108182025-04-30T06:30:16.229ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
108192025-04-30T06:30:16.233ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be083cf2-1451-4403-a2d3-b0f92ad53557", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108202025-04-30T06:30:16.233ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
108212025-04-30T06:30:16.235ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93744303-7f3a-4da9-89b8-667074b324e6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108222025-04-30T06:30:16.236ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
108232025-04-30T06:30:16.237ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e3d43a4-cbcd-4a8c-ae0e-a8dbbf077722", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108242025-04-30T06:30:16.238ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
108252025-04-30T06:30:16.239ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4ab99a8-1089-480b-9d84-a759bbc67d0b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108262025-04-30T06:30:16.239ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
108272025-04-30T06:30:16.240ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2e77807-46b2-4c1f-9745-b4e46c7a9e63", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108282025-04-30T06:30:16.240ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
108292025-04-30T06:30:16.241ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bfeda5e-11f7-4e8e-9d6b-20cb676ecd0f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108302025-04-30T06:30:16.242ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
108312025-04-30T06:30:16.244ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc8272b5-f02d-454b-bfd8-d8338365a944", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108322025-04-30T06:30:16.244ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
108332025-04-30T06:30:16.246ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1d4381e-52ec-4606-ae38-7dc093aaff7a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108342025-04-30T06:30:16.246ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
108352025-04-30T06:30:16.250ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3043aeb-d3a7-4b30-8728-8fa0db0a4ad4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108362025-04-30T06:30:16.250ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
108372025-04-30T06:30:16.252ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "533dfb19-07c8-45d6-9bbe-2f59e5abec81", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108382025-04-30T06:30:16.252ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
108392025-04-30T06:30:16.254ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8326f44-97fd-49aa-966a-fa2bbe4feb69", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108402025-04-30T06:30:16.254ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
108412025-04-30T06:30:16.258ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d10046cd-f536-4b2d-b0d4-1d081688bc96", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108422025-04-30T06:30:16.258ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
108432025-04-30T06:30:16.258ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
108442025-04-30T06:30:16.259ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08019ae9-0f75-41b1-abec-e84fab8bf856", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108452025-04-30T06:30:16.259ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
108462025-04-30T06:30:16.261ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fbf51cf-dc0d-419b-a66a-3ed5f69c8b8f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108472025-04-30T06:30:16.261ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
108482025-04-30T06:30:16.262ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee598603-f2d4-47a7-9116-266aa876ced6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108492025-04-30T06:30:16.262ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
108502025-04-30T06:30:16.263ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "493b2d00-c432-4375-8c2b-efbfdd74c07d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108512025-04-30T06:30:16.263ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
108522025-04-30T06:30:16.263ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108532025-04-30T06:30:16.263ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108542025-04-30T06:30:16.263ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108552025-04-30T06:30:16.264ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108562025-04-30T06:30:16.267ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108572025-04-30T06:30:16.270ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
108582025-04-30T06:30:16.270ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
108592025-04-30T06:30:16.270ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108602025-04-30T06:30:16.270ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108612025-04-30T06:30:16.270ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108622025-04-30T06:30:16.270ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108632025-04-30T06:30:16.283ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108642025-04-30T06:30:16.310ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
108652025-04-30T06:30:16.310ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108662025-04-30T06:30:16.310ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108672025-04-30T06:30:16.311ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108682025-04-30T06:30:16.312ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
108692025-04-30T06:30:16.312ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108702025-04-30T06:30:16.312ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108712025-04-30T06:30:16.312ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
108722025-04-30T06:30:16.313ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108732025-04-30T06:30:16.415ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
108742025-04-30T06:30:16.415ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
108752025-04-30T06:30:16.416ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:64402 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
108762025-04-30T06:30:16.418ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:64402 req_id = 751b1b20-1e9f-4c5b-81d2-c84e5177f716 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
108772025-04-30T06:30:16.419ZINFOtest_image_promotion (dropshot): request completed latency_us = 1858 local_addr = [::1]:60070 method = GET remote_addr = [::1]:64402 req_id = 751b1b20-1e9f-4c5b-81d2-c84e5177f716 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
108782025-04-30T06:30:16.419ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108792025-04-30T06:30:16.431ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
108802025-04-30T06:30:16.431ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108812025-04-30T06:30:16.431ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
108822025-04-30T06:30:16.431ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108832025-04-30T06:30:16.432ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "751b1b20-1e9f-4c5b-81d2-c84e5177f716", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:15 GMT"} })
108842025-04-30T06:30:16.434ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
108852025-04-30T06:30:16.434ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
108862025-04-30T06:30:16.434ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
108872025-04-30T06:30:16.436ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
108882025-04-30T06:30:16.436ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108892025-04-30T06:30:16.436ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108902025-04-30T06:30:16.436ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108912025-04-30T06:30:16.436ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108922025-04-30T06:30:16.441ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108932025-04-30T06:30:16.441ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
108942025-04-30T06:30:16.442ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108952025-04-30T06:30:16.444ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108962025-04-30T06:30:16.449ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
108972025-04-30T06:30:16.449ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108982025-04-30T06:30:16.449ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108992025-04-30T06:30:16.457ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109002025-04-30T06:30:16.492ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
109012025-04-30T06:30:16.492ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
109022025-04-30T06:30:16.492ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
109032025-04-30T06:30:16.493ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
109042025-04-30T06:30:16.493ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
109052025-04-30T06:30:16.493ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
109062025-04-30T06:30:16.493ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
109072025-04-30T06:30:16.494ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
109082025-04-30T06:30:16.494ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
109092025-04-30T06:30:16.500ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
109102025-04-30T06:30:16.500ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 230.499\u{b5}s iteration = 3 status = {"error":"task disabled"}
109112025-04-30T06:30:16.500ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
109122025-04-30T06:30:16.500ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
109132025-04-30T06:30:16.501ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109142025-04-30T06:30:16.501ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109152025-04-30T06:30:16.501ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109162025-04-30T06:30:16.501ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109172025-04-30T06:30:16.502ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
109182025-04-30T06:30:16.502ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
109192025-04-30T06:30:16.502ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109202025-04-30T06:30:16.502ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
109212025-04-30T06:30:16.502ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
109222025-04-30T06:30:16.503ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
109232025-04-30T06:30:16.503ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
109242025-04-30T06:30:16.503ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109252025-04-30T06:30:16.503ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109262025-04-30T06:30:16.504ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109272025-04-30T06:30:16.504ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
109282025-04-30T06:30:16.506ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
109292025-04-30T06:30:16.510ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 15.865611ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
109302025-04-30T06:30:16.510ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109312025-04-30T06:30:16.516ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
109322025-04-30T06:30:16.516ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109332025-04-30T06:30:16.516ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109342025-04-30T06:30:16.516ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109352025-04-30T06:30:16.519ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109362025-04-30T06:30:16.526ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
109372025-04-30T06:30:16.526ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
109382025-04-30T06:30:16.526ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
109392025-04-30T06:30:16.527ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
109402025-04-30T06:30:16.527ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
109412025-04-30T06:30:16.527ZINFOtest_image_promotion (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
109422025-04-30T06:30:16.528ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = instance_updater elapsed = 304.977\u{b5}s iteration = 2 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
109432025-04-30T06:30:16.528ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
109442025-04-30T06:30:16.528ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
109452025-04-30T06:30:16.528ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109462025-04-30T06:30:16.528ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
109472025-04-30T06:30:16.528ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
109482025-04-30T06:30:16.529ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
109492025-04-30T06:30:16.529ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
109502025-04-30T06:30:16.529ZINFOtest_image_promotion (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
109512025-04-30T06:30:16.529ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
109522025-04-30T06:30:16.529ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109532025-04-30T06:30:16.529ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
109542025-04-30T06:30:16.529ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109552025-04-30T06:30:16.530ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109562025-04-30T06:30:16.530ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
109572025-04-30T06:30:16.531ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
109582025-04-30T06:30:16.531ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
109592025-04-30T06:30:16.533ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
109602025-04-30T06:30:16.534ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 518.830035ms iteration = 39 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
109612025-04-30T06:30:16.534ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 40 reason = Signaled
109622025-04-30T06:30:16.534ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
109632025-04-30T06:30:16.534ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109642025-04-30T06:30:16.534ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109652025-04-30T06:30:16.534ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109662025-04-30T06:30:16.535ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109672025-04-30T06:30:16.538ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109682025-04-30T06:30:16.538ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109692025-04-30T06:30:16.538ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109702025-04-30T06:30:16.539ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
109712025-04-30T06:30:16.543ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109722025-04-30T06:30:16.550ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
109732025-04-30T06:30:16.550ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
109742025-04-30T06:30:16.550ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
109752025-04-30T06:30:16.550ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
109762025-04-30T06:30:16.557ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109772025-04-30T06:30:16.562ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
109782025-04-30T06:30:16.563ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
109792025-04-30T06:30:16.563ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109802025-04-30T06:30:16.563ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
109812025-04-30T06:30:16.563ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109822025-04-30T06:30:16.564ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109832025-04-30T06:30:16.566ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109842025-04-30T06:30:16.575ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
109852025-04-30T06:30:16.575ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109862025-04-30T06:30:16.575ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109872025-04-30T06:30:16.575ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109882025-04-30T06:30:16.579ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = c6d3fa1c-8b17-436f-88a0-04f247ccf8ea uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
109892025-04-30T06:30:16.579ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
109902025-04-30T06:30:16.581ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 6d 85 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 m............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56726 req_id = 2c21d4b4-1791-4951-8ab5-77f0b550d1d9
109912025-04-30T06:30:16.581ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28037,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56726 req_id = 2c21d4b4-1791-4951-8ab5-77f0b550d1d9
109922025-04-30T06:30:16.581ZDEBGtest_image_promotion (store): query key key = _dendrite._tcp
109932025-04-30T06:30:16.581ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109942025-04-30T06:30:16.581ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56726 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2c21d4b4-1791-4951-8ab5-77f0b550d1d9
109952025-04-30T06:30:16.582ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 1c c2 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:59706 req_id = 50ce58bf-3ab6-4b3d-b38e-2f9ffde88001
109962025-04-30T06:30:16.582ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7362,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59706 req_id = 50ce58bf-3ab6-4b3d-b38e-2f9ffde88001
109972025-04-30T06:30:16.582ZDEBGtest_image_promotion (store): query key key = _dendrite._tcp
109982025-04-30T06:30:16.582ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109992025-04-30T06:30:16.582ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59706 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 50ce58bf-3ab6-4b3d-b38e-2f9ffde88001
110002025-04-30T06:30:16.582ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = c6d3fa1c-8b17-436f-88a0-04f247ccf8ea resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110012025-04-30T06:30:16.582ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = c6d3fa1c-8b17-436f-88a0-04f247ccf8ea roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110022025-04-30T06:30:16.586ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = c6d3fa1c-8b17-436f-88a0-04f247ccf8ea resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110032025-04-30T06:30:16.586ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110042025-04-30T06:30:16.589ZDEBGtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3481, tv_nsec: 387451845 } })
110052025-04-30T06:30:16.589ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
110062025-04-30T06:30:16.589ZDEBGtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3481, tv_nsec: 388153012 } })
110072025-04-30T06:30:16.589ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
110082025-04-30T06:30:16.589ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 9d bf 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:59744 req_id = ce80e722-3aef-4eb8-bb36-681d02bb6ec7
110092025-04-30T06:30:16.589ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40383,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59744 req_id = ce80e722-3aef-4eb8-bb36-681d02bb6ec7
110102025-04-30T06:30:16.589ZDEBGtest_image_promotion (store): query key key = _dendrite._tcp
110112025-04-30T06:30:16.589ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110122025-04-30T06:30:16.589ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59744 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ce80e722-3aef-4eb8-bb36-681d02bb6ec7
110132025-04-30T06:30:16.594ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110142025-04-30T06:30:16.607ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
110152025-04-30T06:30:16.607ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
110162025-04-30T06:30:16.608ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110172025-04-30T06:30:16.608ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110182025-04-30T06:30:16.609ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
110192025-04-30T06:30:16.609ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
110202025-04-30T06:30:16.609ZDEBGtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3481, tv_nsec: 408859448 } })
110212025-04-30T06:30:16.610ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
110222025-04-30T06:30:16.610ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110232025-04-30T06:30:16.611ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110242025-04-30T06:30:16.611ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110252025-04-30T06:30:16.611ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110262025-04-30T06:30:16.612ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 32862 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = c6d3fa1c-8b17-436f-88a0-04f247ccf8ea response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110272025-04-30T06:30:16.613ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0b 2e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51957 req_id = 9c6818d4-35cc-4f9d-a86d-544f6f1e1b6a
110282025-04-30T06:30:16.614ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2862,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51957 req_id = 9c6818d4-35cc-4f9d-a86d-544f6f1e1b6a
110292025-04-30T06:30:16.614ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110302025-04-30T06:30:16.615ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51957 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9c6818d4-35cc-4f9d-a86d-544f6f1e1b6a
110312025-04-30T06:30:16.615ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 4c 43 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e LC..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63729 req_id = 77681312-6f31-4346-91e1-b03f9aa81782
110322025-04-30T06:30:16.615ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19523,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63729 req_id = 77681312-6f31-4346-91e1-b03f9aa81782
110332025-04-30T06:30:16.615ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110342025-04-30T06:30:16.615ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63729 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 77681312-6f31-4346-91e1-b03f9aa81782
110352025-04-30T06:30:16.615ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f6 e6 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57730 req_id = 33d9f029-9d95-4c7c-a04d-006005b36f9a
110362025-04-30T06:30:16.615ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63206,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57730 req_id = 33d9f029-9d95-4c7c-a04d-006005b36f9a
110372025-04-30T06:30:16.616ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
110382025-04-30T06:30:16.616ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110392025-04-30T06:30:16.616ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57730 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 33d9f029-9d95-4c7c-a04d-006005b36f9a
110402025-04-30T06:30:16.620ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6d3fa1c-8b17-436f-88a0-04f247ccf8ea", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
110412025-04-30T06:30:16.624ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
110422025-04-30T06:30:16.721ZINFOtest_image_promotion (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
110432025-04-30T06:30:16.721ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:60070/vpc-routes
110442025-04-30T06:30:16.722ZINFOtest_image_promotion (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
110452025-04-30T06:30:16.722ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 194.12448ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
110462025-04-30T06:30:16.723ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:33105 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
110472025-04-30T06:30:16.725ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 8da9258c-e4f7-4e43-8b7c-4556b0ed2d64 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
110482025-04-30T06:30:16.726ZINFOtest_image_promotion (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
110492025-04-30T06:30:16.826ZINFOtest_image_promotion (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
110502025-04-30T06:30:16.826ZDEBGtest_image_promotion (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
110512025-04-30T06:30:16.826ZDEBGtest_image_promotion (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
110522025-04-30T06:30:16.826ZWARNtest_image_promotion (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
110532025-04-30T06:30:16.828ZWARNtest_image_promotion (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
110542025-04-30T06:30:16.828ZINFOtest_image_promotion (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
110552025-04-30T06:30:16.890ZINFOtest_image_promotion (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
110562025-04-30T06:30:16.890ZDEBGtest_image_promotion (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
110572025-04-30T06:30:16.891ZDEBGtest_image_promotion (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
110582025-04-30T06:30:16.891ZWARNtest_image_promotion (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
110592025-04-30T06:30:16.892ZWARNtest_image_promotion (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
110602025-04-30T06:30:16.893ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3481, tv_nsec: 692029706 } })
110612025-04-30T06:30:16.893ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
110622025-04-30T06:30:16.893ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 34 e8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 4...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50292 req_id = 6ec01b94-84dc-4d21-82c9-e77bdafc78c6
110632025-04-30T06:30:16.894ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13544,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50292 req_id = 6ec01b94-84dc-4d21-82c9-e77bdafc78c6
110642025-04-30T06:30:16.894ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110652025-04-30T06:30:16.894ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50292 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6ec01b94-84dc-4d21-82c9-e77bdafc78c6
110662025-04-30T06:30:16.894ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 8da9258c-e4f7-4e43-8b7c-4556b0ed2d64 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
110672025-04-30T06:30:16.895ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 8da9258c-e4f7-4e43-8b7c-4556b0ed2d64 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
110682025-04-30T06:30:16.895ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 8da9258c-e4f7-4e43-8b7c-4556b0ed2d64 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
110692025-04-30T06:30:16.897ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
110702025-04-30T06:30:16.897ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110712025-04-30T06:30:16.898ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110722025-04-30T06:30:16.900ZINFOtest_image_promotion (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
110732025-04-30T06:30:16.957ZINFOtest_image_promotion (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
110742025-04-30T06:30:16.957ZDEBGtest_image_promotion (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
110752025-04-30T06:30:16.958ZDEBGtest_image_promotion (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
110762025-04-30T06:30:16.959ZWARNtest_image_promotion (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
110772025-04-30T06:30:16.963ZWARNtest_image_promotion (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
110782025-04-30T06:30:16.964ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:33105 req_id = 56b74001-c5e0-4771-9dc3-7e6ad662e96a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
110792025-04-30T06:30:16.964ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
110802025-04-30T06:30:16.965ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
110812025-04-30T06:30:16.965ZINFOtest_image_promotion (dropshot): request completed latency_us = 890 local_addr = [::1]:60070 method = GET remote_addr = [::1]:33105 req_id = 56b74001-c5e0-4771-9dc3-7e6ad662e96a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
110822025-04-30T06:30:16.967ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
110832025-04-30T06:30:16.968ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
110842025-04-30T06:30:16.974ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 248716 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 8da9258c-e4f7-4e43-8b7c-4556b0ed2d64 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
110852025-04-30T06:30:16.977ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
110862025-04-30T06:30:16.979ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1c f6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50432 req_id = a1cd58ce-39ee-4735-9649-04857edd6cec
110872025-04-30T06:30:16.979ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7414,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50432 req_id = a1cd58ce-39ee-4735-9649-04857edd6cec
110882025-04-30T06:30:16.980ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110892025-04-30T06:30:16.980ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50432 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a1cd58ce-39ee-4735-9649-04857edd6cec
110902025-04-30T06:30:16.980ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "8da9258c-e4f7-4e43-8b7c-4556b0ed2d64", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
110912025-04-30T06:30:16.981ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110922025-04-30T06:30:16.981ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
110932025-04-30T06:30:16.981ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
110942025-04-30T06:30:16.981ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:60070/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "56b74001-c5e0-4771-9dc3-7e6ad662e96a", "content-length": "2", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
110952025-04-30T06:30:16.981ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
110962025-04-30T06:30:16.981ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110972025-04-30T06:30:16.982ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110982025-04-30T06:30:16.986ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
110992025-04-30T06:30:16.986ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
111002025-04-30T06:30:17.086ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
111012025-04-30T06:30:17.087ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
111022025-04-30T06:30:17.089ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111032025-04-30T06:30:17.089ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
111042025-04-30T06:30:17.089ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
111052025-04-30T06:30:17.090ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
111062025-04-30T06:30:17.091ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
111072025-04-30T06:30:17.092ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111082025-04-30T06:30:17.092ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111092025-04-30T06:30:17.092ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
111102025-04-30T06:30:17.092ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111112025-04-30T06:30:17.093ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
111122025-04-30T06:30:17.096ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
111132025-04-30T06:30:17.096ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111142025-04-30T06:30:17.097ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3a fc 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c :............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59259 req_id = 77b64a04-97b4-47e5-a7f1-1a0c652a5411
111152025-04-30T06:30:17.097ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15100,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59259 req_id = 77b64a04-97b4-47e5-a7f1-1a0c652a5411
111162025-04-30T06:30:17.097ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
111172025-04-30T06:30:17.097ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15100, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59259 req_id = 77b64a04-97b4-47e5-a7f1-1a0c652a5411
111182025-04-30T06:30:17.097ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111192025-04-30T06:30:17.101ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
111202025-04-30T06:30:17.101ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
111212025-04-30T06:30:17.101ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111222025-04-30T06:30:17.101ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
111232025-04-30T06:30:17.101ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
111242025-04-30T06:30:17.103ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
111252025-04-30T06:30:17.104ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111262025-04-30T06:30:17.110ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111272025-04-30T06:30:17.110ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111282025-04-30T06:30:17.110ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
111292025-04-30T06:30:17.111ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111302025-04-30T06:30:17.111ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
111312025-04-30T06:30:17.111ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
111322025-04-30T06:30:17.111ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
111332025-04-30T06:30:17.115ZINFOtest_image_promotion (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
111342025-04-30T06:30:17.115ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:60070/eip-gateways
111352025-04-30T06:30:17.116ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = PUT remote_addr = [::1]:33105 req_id = b8584427-590d-40e5-87f4-b7f0bb663d1e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
111362025-04-30T06:30:17.117ZINFOtest_image_promotion (dropshot): request completed latency_us = 515 local_addr = [::1]:60070 method = PUT remote_addr = [::1]:33105 req_id = b8584427-590d-40e5-87f4-b7f0bb663d1e response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
111372025-04-30T06:30:17.117ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111382025-04-30T06:30:17.121ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
111392025-04-30T06:30:17.122ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
111402025-04-30T06:30:17.122ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
111412025-04-30T06:30:17.125ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
111422025-04-30T06:30:17.128ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111432025-04-30T06:30:17.128ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
111442025-04-30T06:30:17.130ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111452025-04-30T06:30:17.134ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:60070/eip-gateways", status: 204, headers: {"x-request-id": "b8584427-590d-40e5-87f4-b7f0bb663d1e", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111462025-04-30T06:30:17.136ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 607.001802ms iteration = 2 status = {}
111472025-04-30T06:30:17.201ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:60070/v2p
111482025-04-30T06:30:17.202ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
111492025-04-30T06:30:17.202ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
111502025-04-30T06:30:17.203ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
111512025-04-30T06:30:17.207ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:34620 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
111522025-04-30T06:30:17.210ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:34620 req_id = 4b69e4de-9d84-45e4-af8f-72fc65d9830f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
111532025-04-30T06:30:17.215ZINFOtest_image_promotion (dropshot): request completed latency_us = 5158 local_addr = [::1]:60070 method = GET remote_addr = [::1]:34620 req_id = 4b69e4de-9d84-45e4-af8f-72fc65d9830f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
111542025-04-30T06:30:17.215ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
111552025-04-30T06:30:17.215ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111562025-04-30T06:30:17.215ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
111572025-04-30T06:30:17.215ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
111582025-04-30T06:30:17.216ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
111592025-04-30T06:30:17.218ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:60070/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b69e4de-9d84-45e4-af8f-72fc65d9830f", "content-length": "2", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111602025-04-30T06:30:17.218ZINFOtest_image_promotion (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
111612025-04-30T06:30:17.218ZINFOtest_image_promotion (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
111622025-04-30T06:30:17.218ZINFOtest_image_promotion (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
111632025-04-30T06:30:17.222ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 719.116893ms iteration = 2 status = {}
111642025-04-30T06:30:17.225ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111652025-04-30T06:30:17.235ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
111662025-04-30T06:30:17.235ZERROtest_image_promotion (BackgroundTasks): ipv6 addresses for service zone nat not implemented background_task = service_zone_nat_tracker
111672025-04-30T06:30:17.236ZERROtest_image_promotion (BackgroundTasks): generated config for fewer than the minimum allowed number of dns zones background_task = service_zone_nat_tracker
111682025-04-30T06:30:17.236ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 735.485046ms iteration = 2 status = {"error":"generated config for fewer than the minimum allowed number of dns zones"}
111692025-04-30T06:30:17.245ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3396ee7c-8ece-414f-8cd0-cc8f2da2d2ef", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111702025-04-30T06:30:17.245ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
111712025-04-30T06:30:17.258ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ee440ca-82b2-4bc8-99da-5e178fe32393", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111722025-04-30T06:30:17.258ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
111732025-04-30T06:30:17.264ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9463245a-61ee-4ecc-996c-7f9d20388271", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111742025-04-30T06:30:17.264ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
111752025-04-30T06:30:17.268ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4baccacc-60da-4eaa-8c0a-72daf8c46066", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111762025-04-30T06:30:17.268ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
111772025-04-30T06:30:17.272ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0ccc348-21b6-4d9e-889d-1abe1251d557", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111782025-04-30T06:30:17.272ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
111792025-04-30T06:30:17.275ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a463094f-d22e-4b94-9065-0aaf08952a22", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111802025-04-30T06:30:17.275ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
111812025-04-30T06:30:17.283ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
111822025-04-30T06:30:17.285ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c286627e-d10e-496d-93c7-2f0132a5ac1b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111832025-04-30T06:30:17.285ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
111842025-04-30T06:30:17.289ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35ebb1e1-166f-4efc-848e-a6ac98950012", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111852025-04-30T06:30:17.289ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
111862025-04-30T06:30:17.291ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c45e035-cdff-4a5b-a9f1-b40fb85a4e0d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111872025-04-30T06:30:17.291ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
111882025-04-30T06:30:17.294ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c446d31d-2fa9-4a9b-89ff-b09f829a7839", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111892025-04-30T06:30:17.295ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
111902025-04-30T06:30:17.296ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4a5f4af-6b9a-4d9f-b3f3-45b7c715600c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111912025-04-30T06:30:17.297ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
111922025-04-30T06:30:17.298ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31385590-479b-466a-b3ca-a30df50d28f0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111932025-04-30T06:30:17.298ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
111942025-04-30T06:30:17.301ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a79f7f4-89e4-4aff-81f6-1eb7993ce7e2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111952025-04-30T06:30:17.301ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
111962025-04-30T06:30:17.305ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "45d1046b-6f98-4830-85ea-db6aadf30428", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111972025-04-30T06:30:17.305ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
111982025-04-30T06:30:17.307ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f265d72e-7e37-4610-98fd-48584a819d30", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
111992025-04-30T06:30:17.308ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
112002025-04-30T06:30:17.309ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57a04721-104f-4292-883d-657ee806c53d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112012025-04-30T06:30:17.309ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
112022025-04-30T06:30:17.311ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66bd6a36-37c8-4982-9b6a-91f4c159212e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112032025-04-30T06:30:17.312ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
112042025-04-30T06:30:17.315ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "060c8bd4-4961-426f-8e9f-dfe6913d643a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112052025-04-30T06:30:17.315ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
112062025-04-30T06:30:17.319ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "533aa9ae-b2f3-41a4-8d1d-8d612cbe0aa8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112072025-04-30T06:30:17.319ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
112082025-04-30T06:30:17.320ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5064a438-240b-4dce-939b-20fba0ccefd6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112092025-04-30T06:30:17.321ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
112102025-04-30T06:30:17.322ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2716b80b-4017-4b79-bdd3-2b80c750a503", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112112025-04-30T06:30:17.322ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
112122025-04-30T06:30:17.323ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a7e3d5a-f888-4f15-add5-17a7c0aa03e0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112132025-04-30T06:30:17.324ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
112142025-04-30T06:30:17.324ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "998d0746-495a-45bc-a0de-ccfe312915b1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112152025-04-30T06:30:17.325ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
112162025-04-30T06:30:17.326ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d70933fd-bb8d-404f-b3cc-9d9107448de4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112172025-04-30T06:30:17.326ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
112182025-04-30T06:30:17.328ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "928cfe19-4574-49b7-9cd5-f4607f3b626b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112192025-04-30T06:30:17.328ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
112202025-04-30T06:30:17.331ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5580e79-2e2d-44a1-8212-866c7ebc3b84", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112212025-04-30T06:30:17.332ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
112222025-04-30T06:30:17.333ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112232025-04-30T06:30:17.333ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112242025-04-30T06:30:17.333ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112252025-04-30T06:30:17.334ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
112262025-04-30T06:30:17.334ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112272025-04-30T06:30:17.336ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d308268-597e-437b-80e9-23715fca881d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112282025-04-30T06:30:17.336ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
112292025-04-30T06:30:17.340ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112302025-04-30T06:30:17.348ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112312025-04-30T06:30:17.348ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112322025-04-30T06:30:17.348ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
112332025-04-30T06:30:17.349ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112342025-04-30T06:30:17.352ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9664834-6928-449f-b094-a16a6223a969", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112352025-04-30T06:30:17.352ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
112362025-04-30T06:30:17.354ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112372025-04-30T06:30:17.354ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
112382025-04-30T06:30:17.355ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112392025-04-30T06:30:17.360ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d7d3a0a-de6c-47c1-85ee-24446127e25a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112402025-04-30T06:30:17.360ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
112412025-04-30T06:30:17.366ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac47c6ce-196f-4d05-8f19-4b5a35c9791a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112422025-04-30T06:30:17.366ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
112432025-04-30T06:30:17.367ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03187a1f-2fb0-48f3-937d-bf4e3349e469", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112442025-04-30T06:30:17.367ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
112452025-04-30T06:30:17.368ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a34f41d-de2b-483e-a9da-b453cfcbce25", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112462025-04-30T06:30:17.368ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
112472025-04-30T06:30:17.370ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5574bac-b741-496c-ae44-da5dbd1b5b37", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112482025-04-30T06:30:17.370ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
112492025-04-30T06:30:17.371ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82b46de2-4498-4232-9d4e-88b7aa113541", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112502025-04-30T06:30:17.371ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
112512025-04-30T06:30:17.373ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f5b70b1-f5ac-4855-9467-01573b4e7b3b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112522025-04-30T06:30:17.373ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
112532025-04-30T06:30:17.378ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "999d7fe6-72c5-44c2-80d6-db34bd61b57c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112542025-04-30T06:30:17.378ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
112552025-04-30T06:30:17.380ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ace78a6a-29d8-4d6d-b236-51a02b6adc19", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112562025-04-30T06:30:17.380ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
112572025-04-30T06:30:17.385ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e54f6a61-3be1-4dc6-a99d-7dbe33ebcfb3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112582025-04-30T06:30:17.385ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
112592025-04-30T06:30:17.388ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1fb0724-fc8e-48d2-93f2-b7d2586b3a12", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112602025-04-30T06:30:17.388ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
112612025-04-30T06:30:17.392ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c9a0ea1-17cb-4768-8e7a-b88eb89b64f7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112622025-04-30T06:30:17.393ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
112632025-04-30T06:30:17.393ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
112642025-04-30T06:30:17.397ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b100bf7d-e460-41ea-935a-4470cddad05b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112652025-04-30T06:30:17.397ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
112662025-04-30T06:30:17.398ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8e7fb5e-3401-43b1-b7c6-b1fd65a8a3c8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112672025-04-30T06:30:17.399ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
112682025-04-30T06:30:17.400ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98370072-5598-439e-8570-e3a0b7819863", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112692025-04-30T06:30:17.401ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
112702025-04-30T06:30:17.404ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fd1a4bc-8a78-4c33-bfde-b4a62710e3d0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112712025-04-30T06:30:17.404ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
112722025-04-30T06:30:17.404ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89803250-bb4d-4e8f-8ec1-ee14f17f8b42", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
112732025-04-30T06:30:17.404ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
112742025-04-30T06:30:17.405ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112752025-04-30T06:30:17.405ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112762025-04-30T06:30:17.405ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112772025-04-30T06:30:17.406ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112782025-04-30T06:30:17.434ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112792025-04-30T06:30:17.438ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
112802025-04-30T06:30:17.438ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
112812025-04-30T06:30:17.438ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112822025-04-30T06:30:17.438ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112832025-04-30T06:30:17.438ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112842025-04-30T06:30:17.439ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112852025-04-30T06:30:17.443ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112862025-04-30T06:30:17.447ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
112872025-04-30T06:30:17.448ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112882025-04-30T06:30:17.448ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112892025-04-30T06:30:17.451ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112902025-04-30T06:30:17.453ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112912025-04-30T06:30:17.453ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112922025-04-30T06:30:17.453ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112932025-04-30T06:30:17.453ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
112942025-04-30T06:30:17.454ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112952025-04-30T06:30:17.577ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
112962025-04-30T06:30:17.577ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
112972025-04-30T06:30:17.584ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:38092 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
112982025-04-30T06:30:17.585ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
112992025-04-30T06:30:17.585ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113002025-04-30T06:30:17.585ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113012025-04-30T06:30:17.594ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113022025-04-30T06:30:17.594ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113032025-04-30T06:30:17.594ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
113042025-04-30T06:30:17.594ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113052025-04-30T06:30:17.594ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:38092 req_id = 09c2f4dc-51eb-407e-adb4-e67056a35323 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
113062025-04-30T06:30:17.596ZINFOtest_image_promotion (dropshot): request completed latency_us = 1636 local_addr = [::1]:60070 method = GET remote_addr = [::1]:38092 req_id = 09c2f4dc-51eb-407e-adb4-e67056a35323 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
113072025-04-30T06:30:17.597ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "09c2f4dc-51eb-407e-adb4-e67056a35323", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:16 GMT"} })
113082025-04-30T06:30:17.608ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
113092025-04-30T06:30:17.608ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
113102025-04-30T06:30:17.608ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
113112025-04-30T06:30:17.621ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
113122025-04-30T06:30:17.621ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113132025-04-30T06:30:17.621ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113142025-04-30T06:30:17.621ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113152025-04-30T06:30:17.622ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113162025-04-30T06:30:17.622ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113172025-04-30T06:30:17.622ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113182025-04-30T06:30:17.622ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113192025-04-30T06:30:17.622ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113202025-04-30T06:30:17.630ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113212025-04-30T06:30:17.631ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
113222025-04-30T06:30:17.631ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113232025-04-30T06:30:17.639ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113242025-04-30T06:30:17.662ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
113252025-04-30T06:30:17.695ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113262025-04-30T06:30:17.695ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113272025-04-30T06:30:17.695ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113282025-04-30T06:30:17.723ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
113292025-04-30T06:30:17.723ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
113302025-04-30T06:30:17.723ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.189670321s iteration = 40 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
113312025-04-30T06:30:17.724ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 41 reason = Signaled
113322025-04-30T06:30:17.724ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
113332025-04-30T06:30:17.724ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113342025-04-30T06:30:17.724ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113352025-04-30T06:30:17.724ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113362025-04-30T06:30:17.725ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113372025-04-30T06:30:17.734ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113382025-04-30T06:30:17.738ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
113392025-04-30T06:30:17.739ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113402025-04-30T06:30:17.739ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113412025-04-30T06:30:17.739ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113422025-04-30T06:30:17.740ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
113432025-04-30T06:30:17.742ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
113442025-04-30T06:30:17.742ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
113452025-04-30T06:30:17.743ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 3b 98 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ;............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53916 req_id = b059de64-1174-454c-94d1-e4f6e5345753
113462025-04-30T06:30:17.743ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15256,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53916 req_id = b059de64-1174-454c-94d1-e4f6e5345753
113472025-04-30T06:30:17.744ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
113482025-04-30T06:30:17.744ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113492025-04-30T06:30:17.744ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53916 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b059de64-1174-454c-94d1-e4f6e5345753
113502025-04-30T06:30:17.744ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3482, tv_nsec: 543611859 } })
113512025-04-30T06:30:17.744ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
113522025-04-30T06:30:17.745ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1c 59 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .Y..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49740 req_id = f717c5c9-9e41-461d-9bcd-cb1b35dd1645
113532025-04-30T06:30:17.745ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7257,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49740 req_id = f717c5c9-9e41-461d-9bcd-cb1b35dd1645
113542025-04-30T06:30:17.745ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113552025-04-30T06:30:17.745ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49740 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f717c5c9-9e41-461d-9bcd-cb1b35dd1645
113562025-04-30T06:30:17.860ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
113572025-04-30T06:30:17.861ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113582025-04-30T06:30:17.861ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113592025-04-30T06:30:17.862ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113602025-04-30T06:30:17.862ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
113612025-04-30T06:30:17.865ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113622025-04-30T06:30:17.865ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
113632025-04-30T06:30:17.865ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
113642025-04-30T06:30:17.866ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
113652025-04-30T06:30:17.867ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
113662025-04-30T06:30:17.867ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ee cf 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55405 req_id = db837830-3ac1-4533-a546-86aef4d142e7
113672025-04-30T06:30:17.867ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61135,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55405 req_id = db837830-3ac1-4533-a546-86aef4d142e7
113682025-04-30T06:30:17.868ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
113692025-04-30T06:30:17.868ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61135, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55405 req_id = db837830-3ac1-4533-a546-86aef4d142e7
113702025-04-30T06:30:17.868ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
113712025-04-30T06:30:17.868ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
113722025-04-30T06:30:17.868ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
113732025-04-30T06:30:17.869ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113742025-04-30T06:30:17.897ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113752025-04-30T06:30:17.897ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113762025-04-30T06:30:17.897ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
113772025-04-30T06:30:17.902ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113782025-04-30T06:30:17.904ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
113792025-04-30T06:30:17.904ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
113802025-04-30T06:30:17.907ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd45f630-d2c3-4acb-b016-540bb5ac6add", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
113812025-04-30T06:30:17.907ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
113822025-04-30T06:30:17.908ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113832025-04-30T06:30:17.908ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
113842025-04-30T06:30:17.908ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113852025-04-30T06:30:17.909ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
113862025-04-30T06:30:17.909ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
113872025-04-30T06:30:17.918ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ec1eb4c-1f1a-4df7-ae60-9a02665c69d4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
113882025-04-30T06:30:17.919ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
113892025-04-30T06:30:17.944ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97fee026-9a1b-4bbb-be08-b8902e117365", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
113902025-04-30T06:30:17.944ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
113912025-04-30T06:30:17.947ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06a39d2c-a78c-49f2-92e4-942017316f87", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
113922025-04-30T06:30:17.947ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
113932025-04-30T06:30:17.985ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
113942025-04-30T06:30:17.985ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
113952025-04-30T06:30:18.024ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3ec5cb3-5254-4509-b885-36ceb5175839", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
113962025-04-30T06:30:18.024ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
113972025-04-30T06:30:18.029ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03f6266d-bda9-404b-9926-65a5e8a02f59", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
113982025-04-30T06:30:18.029ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
113992025-04-30T06:30:18.033ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f70aa292-4984-4f14-8190-fbca1224619d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114002025-04-30T06:30:18.033ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
114012025-04-30T06:30:18.035ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65fca11a-a0a1-434d-84c8-85ce47107624", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114022025-04-30T06:30:18.045ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
114032025-04-30T06:30:18.045ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114042025-04-30T06:30:18.045ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114052025-04-30T06:30:18.045ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114062025-04-30T06:30:18.045ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
114072025-04-30T06:30:18.047ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114082025-04-30T06:30:18.048ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59f63a79-51c2-4752-ae06-e61ae11d0828", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114092025-04-30T06:30:18.049ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
114102025-04-30T06:30:18.051ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114112025-04-30T06:30:18.056ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114122025-04-30T06:30:18.056ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114132025-04-30T06:30:18.056ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
114142025-04-30T06:30:18.056ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114152025-04-30T06:30:18.058ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f23501f0-2928-4d32-86cb-fed4538ac162", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114162025-04-30T06:30:18.058ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
114172025-04-30T06:30:18.061ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7322746e-ae54-42b1-bbd8-b922d11ab5b8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114182025-04-30T06:30:18.061ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
114192025-04-30T06:30:18.061ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114202025-04-30T06:30:18.062ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
114212025-04-30T06:30:18.062ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114222025-04-30T06:30:18.063ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61cad318-ff91-49b9-b8fe-2a5477191da6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114232025-04-30T06:30:18.064ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
114242025-04-30T06:30:18.065ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76809100-d4e0-4385-be92-39cceb453989", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114252025-04-30T06:30:18.067ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
114262025-04-30T06:30:18.079ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b94dd5a3-2e13-4036-9231-399b449877fb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114272025-04-30T06:30:18.079ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
114282025-04-30T06:30:18.085ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "30f4b180-2885-45cb-8ef6-066a31764b23", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114292025-04-30T06:30:18.085ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
114302025-04-30T06:30:18.092ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3306367f-72af-489d-af90-52dfc1ef42fd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114312025-04-30T06:30:18.093ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
114322025-04-30T06:30:18.095ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de609f38-4cf9-4af5-b90f-2c540a86a942", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114332025-04-30T06:30:18.095ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
114342025-04-30T06:30:18.098ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b85563b0-6647-4e71-b299-669177128e1d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114352025-04-30T06:30:18.098ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
114362025-04-30T06:30:18.101ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
114372025-04-30T06:30:18.103ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f5cd5e4-f91c-4cfb-bd39-70b207434434", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114382025-04-30T06:30:18.103ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
114392025-04-30T06:30:18.104ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a85d83f6-1f58-46bd-a86c-f89c4da748a1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114402025-04-30T06:30:18.104ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
114412025-04-30T06:30:18.110ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8f62d70-9b4d-43fd-91c0-4818fdfaf837", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114422025-04-30T06:30:18.112ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
114432025-04-30T06:30:18.113ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "593f2b3d-c53c-426d-8c97-9a60f9963dd9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114442025-04-30T06:30:18.114ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
114452025-04-30T06:30:18.120ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aba3e8fa-38c7-43e5-a082-8f3334cde4b2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114462025-04-30T06:30:18.120ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
114472025-04-30T06:30:18.122ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc0b49c9-f3df-4c7e-a324-ddacc1dd1a5b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114482025-04-30T06:30:18.123ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
114492025-04-30T06:30:18.127ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6db31cb7-4681-423d-87a9-b2b3202d94d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114502025-04-30T06:30:18.128ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
114512025-04-30T06:30:18.131ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87e5de68-cb13-4a8a-9913-690d8e29a445", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114522025-04-30T06:30:18.131ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
114532025-04-30T06:30:18.134ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f5aabec-704a-4275-932e-f567a1175b56", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114542025-04-30T06:30:18.135ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
114552025-04-30T06:30:18.137ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "117f055d-0742-4507-97bf-4a27d068acd6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114562025-04-30T06:30:18.137ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
114572025-04-30T06:30:18.142ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff1833d3-0114-4613-8c70-89a35df33c8d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114582025-04-30T06:30:18.142ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
114592025-04-30T06:30:18.146ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6fb56dc-6362-45bd-b0d4-4bc521146584", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114602025-04-30T06:30:18.146ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
114612025-04-30T06:30:18.152ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114622025-04-30T06:30:18.154ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114632025-04-30T06:30:18.154ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114642025-04-30T06:30:18.155ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
114652025-04-30T06:30:18.156ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114662025-04-30T06:30:18.157ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0143882-996d-40e1-9ae2-bca1bc7d6a20", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114672025-04-30T06:30:18.157ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
114682025-04-30T06:30:18.160ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a66ae66-d49a-4f3f-b75c-dec7cd4b2488", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114692025-04-30T06:30:18.160ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
114702025-04-30T06:30:18.161ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114712025-04-30T06:30:18.171ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114722025-04-30T06:30:18.171ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114732025-04-30T06:30:18.171ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
114742025-04-30T06:30:18.172ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114752025-04-30T06:30:18.175ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114762025-04-30T06:30:18.175ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
114772025-04-30T06:30:18.176ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114782025-04-30T06:30:18.176ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d760153-f7bb-428b-a58f-ac727203c2ed", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114792025-04-30T06:30:18.176ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
114802025-04-30T06:30:18.182ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da5ba69b-1b26-43d9-a029-5ad2f104e152", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114812025-04-30T06:30:18.182ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
114822025-04-30T06:30:18.184ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3388c62-60d2-404f-b8bd-df53f0368955", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114832025-04-30T06:30:18.184ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
114842025-04-30T06:30:18.194ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc0dc717-d62a-4d17-b0c3-f6cd5bd9c01c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114852025-04-30T06:30:18.195ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
114862025-04-30T06:30:18.219ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96cf750d-0e20-403d-bbe1-c8705fc310cb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114872025-04-30T06:30:18.219ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
114882025-04-30T06:30:18.227ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c170cb9a-e58c-404f-9ee0-7d20cbc53133", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114892025-04-30T06:30:18.227ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
114902025-04-30T06:30:18.229ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bb008ff-0340-43d0-8274-77657f8a9a74", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114912025-04-30T06:30:18.230ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
114922025-04-30T06:30:18.233ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
114932025-04-30T06:30:18.237ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a659c9d4-bee0-4a39-9bca-9f0f04b70d74", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114942025-04-30T06:30:18.237ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
114952025-04-30T06:30:18.241ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39aae7af-ce20-4aa4-a4f2-7c21c7037f0f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114962025-04-30T06:30:18.241ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
114972025-04-30T06:30:18.242ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52c92efd-d86d-4d57-a53a-7e181b939baf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
114982025-04-30T06:30:18.243ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
114992025-04-30T06:30:18.245ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "024de440-d2d0-4288-be64-cdaff3cb6f76", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
115002025-04-30T06:30:18.245ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
115012025-04-30T06:30:18.247ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bdccc3e-f8cc-4c05-b10b-860b3989eda7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
115022025-04-30T06:30:18.247ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
115032025-04-30T06:30:18.249ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef4a6384-6606-4868-ad5d-e043f2c7470f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
115042025-04-30T06:30:18.250ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
115052025-04-30T06:30:18.250ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115062025-04-30T06:30:18.251ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115072025-04-30T06:30:18.251ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115082025-04-30T06:30:18.252ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115092025-04-30T06:30:18.255ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115102025-04-30T06:30:18.259ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
115112025-04-30T06:30:18.259ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
115122025-04-30T06:30:18.259ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115132025-04-30T06:30:18.259ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115142025-04-30T06:30:18.259ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115152025-04-30T06:30:18.260ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115162025-04-30T06:30:18.265ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115172025-04-30T06:30:18.268ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
115182025-04-30T06:30:18.268ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115192025-04-30T06:30:18.268ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115202025-04-30T06:30:18.269ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115212025-04-30T06:30:18.340ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
115222025-04-30T06:30:18.344ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
115232025-04-30T06:30:18.345ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115242025-04-30T06:30:18.345ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115252025-04-30T06:30:18.345ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115262025-04-30T06:30:18.345ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
115272025-04-30T06:30:18.345ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115282025-04-30T06:30:18.345ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:47659 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
115292025-04-30T06:30:18.345ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:47659 req_id = 59292f24-faed-48c2-894c-601e914fc3e1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
115302025-04-30T06:30:18.345ZINFOtest_image_promotion (dropshot): request completed latency_us = 518 local_addr = [::1]:60070 method = GET remote_addr = [::1]:47659 req_id = 59292f24-faed-48c2-894c-601e914fc3e1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
115312025-04-30T06:30:18.346ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "59292f24-faed-48c2-894c-601e914fc3e1", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
115322025-04-30T06:30:18.352ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
115332025-04-30T06:30:18.352ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
115342025-04-30T06:30:18.352ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
115352025-04-30T06:30:18.356ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
115362025-04-30T06:30:18.356ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115372025-04-30T06:30:18.356ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115382025-04-30T06:30:18.356ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115392025-04-30T06:30:18.356ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115402025-04-30T06:30:18.357ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115412025-04-30T06:30:18.364ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115422025-04-30T06:30:18.365ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115432025-04-30T06:30:18.365ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
115442025-04-30T06:30:18.366ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115452025-04-30T06:30:18.373ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115462025-04-30T06:30:18.379ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
115472025-04-30T06:30:18.379ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115482025-04-30T06:30:18.379ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115492025-04-30T06:30:18.380ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115502025-04-30T06:30:18.398ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115512025-04-30T06:30:18.398ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
115522025-04-30T06:30:18.400ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115532025-04-30T06:30:18.436ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
115542025-04-30T06:30:18.436ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
115552025-04-30T06:30:18.436ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 712.425211ms iteration = 41 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
115562025-04-30T06:30:18.436ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 42 reason = Signaled
115572025-04-30T06:30:18.436ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
115582025-04-30T06:30:18.436ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115592025-04-30T06:30:18.436ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115602025-04-30T06:30:18.436ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115612025-04-30T06:30:18.437ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115622025-04-30T06:30:18.437ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
115632025-04-30T06:30:18.445ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115642025-04-30T06:30:18.450ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
115652025-04-30T06:30:18.450ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115662025-04-30T06:30:18.450ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115672025-04-30T06:30:18.450ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115682025-04-30T06:30:18.453ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
115692025-04-30T06:30:18.453ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
115702025-04-30T06:30:18.454ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4b ac 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 K............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51248 req_id = 9278309a-8c49-49a8-9091-6442982caaed
115712025-04-30T06:30:18.454ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19372,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51248 req_id = 9278309a-8c49-49a8-9091-6442982caaed
115722025-04-30T06:30:18.454ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
115732025-04-30T06:30:18.454ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115742025-04-30T06:30:18.455ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51248 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9278309a-8c49-49a8-9091-6442982caaed
115752025-04-30T06:30:18.455ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3483, tv_nsec: 254261421 } })
115762025-04-30T06:30:18.455ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
115772025-04-30T06:30:18.455ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 18 fe 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51598 req_id = e89b7bbd-1208-430b-8d0f-bd4c795ebdb2
115782025-04-30T06:30:18.455ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6398,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51598 req_id = e89b7bbd-1208-430b-8d0f-bd4c795ebdb2
115792025-04-30T06:30:18.456ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115802025-04-30T06:30:18.456ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51598 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e89b7bbd-1208-430b-8d0f-bd4c795ebdb2
115812025-04-30T06:30:18.506ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
115822025-04-30T06:30:18.507ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115832025-04-30T06:30:18.507ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115842025-04-30T06:30:18.507ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115852025-04-30T06:30:18.507ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
115862025-04-30T06:30:18.508ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115872025-04-30T06:30:18.508ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
115882025-04-30T06:30:18.509ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
115892025-04-30T06:30:18.509ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fb15d354-ca83-4d72-a992-b9993cfe0fdc uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
115902025-04-30T06:30:18.509ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 0e 02 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57878 req_id = 140ed241-2050-40cc-8b8a-46754100d4f2
115912025-04-30T06:30:18.509ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3586,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57878 req_id = 140ed241-2050-40cc-8b8a-46754100d4f2
115922025-04-30T06:30:18.509ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
115932025-04-30T06:30:18.510ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3586, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57878 req_id = 140ed241-2050-40cc-8b8a-46754100d4f2
115942025-04-30T06:30:18.510ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fb15d354-ca83-4d72-a992-b9993cfe0fdc resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
115952025-04-30T06:30:18.510ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fb15d354-ca83-4d72-a992-b9993cfe0fdc roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
115962025-04-30T06:30:18.510ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fb15d354-ca83-4d72-a992-b9993cfe0fdc resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
115972025-04-30T06:30:18.511ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115982025-04-30T06:30:18.519ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115992025-04-30T06:30:18.519ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116002025-04-30T06:30:18.519ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
116012025-04-30T06:30:18.519ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116022025-04-30T06:30:18.520ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
116032025-04-30T06:30:18.520ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
116042025-04-30T06:30:18.520ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
116052025-04-30T06:30:18.523ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 14129 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fb15d354-ca83-4d72-a992-b9993cfe0fdc response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
116062025-04-30T06:30:18.525ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb15d354-ca83-4d72-a992-b9993cfe0fdc", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116072025-04-30T06:30:18.525ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
116082025-04-30T06:30:18.526ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116092025-04-30T06:30:18.526ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
116102025-04-30T06:30:18.527ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116112025-04-30T06:30:18.528ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = da5c73f1-0188-4429-b20d-322fc0cf0520 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
116122025-04-30T06:30:18.528ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = da5c73f1-0188-4429-b20d-322fc0cf0520 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
116132025-04-30T06:30:18.529ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = da5c73f1-0188-4429-b20d-322fc0cf0520 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
116142025-04-30T06:30:18.530ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = da5c73f1-0188-4429-b20d-322fc0cf0520 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
116152025-04-30T06:30:18.532ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfa86445-520d-48bf-b552-0e52e59923f1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116162025-04-30T06:30:18.532ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
116172025-04-30T06:30:18.534ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebb0e704-3a9c-4df8-adee-7196f3653159", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116182025-04-30T06:30:18.534ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
116192025-04-30T06:30:18.535ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 7090 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = da5c73f1-0188-4429-b20d-322fc0cf0520 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
116202025-04-30T06:30:18.536ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "da5c73f1-0188-4429-b20d-322fc0cf0520", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116212025-04-30T06:30:18.536ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116222025-04-30T06:30:18.536ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
116232025-04-30T06:30:18.536ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
116242025-04-30T06:30:18.538ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116252025-04-30T06:30:18.538ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d341a718-cf1b-4590-a9e8-246d959adae7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116262025-04-30T06:30:18.538ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
116272025-04-30T06:30:18.538ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116282025-04-30T06:30:18.540ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116292025-04-30T06:30:18.541ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116302025-04-30T06:30:18.541ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116312025-04-30T06:30:18.541ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116322025-04-30T06:30:18.543ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebf0580d-37db-4043-b825-14d53161176d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116332025-04-30T06:30:18.543ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
116342025-04-30T06:30:18.546ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6296c02c-5d61-4439-8d59-d144d66941c4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116352025-04-30T06:30:18.546ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
116362025-04-30T06:30:18.551ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afb2663b-7ce7-43bc-a4ce-b106ffca5bf4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116372025-04-30T06:30:18.551ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
116382025-04-30T06:30:18.557ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c03e4541-40eb-4863-ba5c-806847930617", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116392025-04-30T06:30:18.557ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
116402025-04-30T06:30:18.562ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "29661271-8087-47e6-998d-3b513b885d61", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116412025-04-30T06:30:18.562ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
116422025-04-30T06:30:18.566ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f75bd830-e04c-4410-9957-14eaa83c1b49", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116432025-04-30T06:30:18.566ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
116442025-04-30T06:30:18.567ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
116452025-04-30T06:30:18.569ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15bbf273-a9b3-4627-8ab0-f173b031cd7c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116462025-04-30T06:30:18.569ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
116472025-04-30T06:30:18.572ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccdc3ade-c17c-4ffa-b5b5-c929dd033bdc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116482025-04-30T06:30:18.572ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
116492025-04-30T06:30:18.574ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0af7b82c-b98a-4e0b-8369-e50e7a3aeb3a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116502025-04-30T06:30:18.575ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
116512025-04-30T06:30:18.576ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3795a66c-270a-428a-9608-b75a0ebab4c5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116522025-04-30T06:30:18.576ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
116532025-04-30T06:30:18.578ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e26e3e4-37d1-437c-b1be-1e680bf1fed8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116542025-04-30T06:30:18.578ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
116552025-04-30T06:30:18.580ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "783e8460-99a0-43bf-8cdc-bdf7fa24ab7c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116562025-04-30T06:30:18.590ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
116572025-04-30T06:30:18.592ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f978989-d1ff-4e9d-a612-db2a1223e4f3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116582025-04-30T06:30:18.592ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
116592025-04-30T06:30:18.594ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2843bfb7-12ab-41df-84a6-accf32debba8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116602025-04-30T06:30:18.594ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
116612025-04-30T06:30:18.596ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf613f55-476a-4161-bd7a-fb00f32dcbbb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116622025-04-30T06:30:18.596ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
116632025-04-30T06:30:18.598ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c1247bc-e996-4255-846b-27bf0e3505d6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116642025-04-30T06:30:18.598ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
116652025-04-30T06:30:18.600ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e94bba9-48ec-42dc-9bc0-95dbff2dc74b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116662025-04-30T06:30:18.600ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
116672025-04-30T06:30:18.601ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d0aa199-7a8f-4d71-9b95-b2eeab4baa9a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116682025-04-30T06:30:18.601ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
116692025-04-30T06:30:18.602ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0f1a605-b225-45cf-87a1-8f09a8f539a8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116702025-04-30T06:30:18.602ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
116712025-04-30T06:30:18.603ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4bfea7e-6786-492a-b81b-fba7c9ab8f40", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116722025-04-30T06:30:18.603ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
116732025-04-30T06:30:18.605ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f96b2b7-26dd-48f8-be12-db4e4430cd41", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116742025-04-30T06:30:18.605ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
116752025-04-30T06:30:18.608ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b66c877d-52eb-43ad-9850-4f9e4a9b321e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116762025-04-30T06:30:18.608ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
116772025-04-30T06:30:18.612ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a1dac51-777d-4a09-b858-4621a4c962c2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116782025-04-30T06:30:18.613ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
116792025-04-30T06:30:18.618ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116802025-04-30T06:30:18.618ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116812025-04-30T06:30:18.618ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116822025-04-30T06:30:18.618ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
116832025-04-30T06:30:18.619ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116842025-04-30T06:30:18.622ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22ef91bc-3ff0-4423-972c-1c428f17d481", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:17 GMT"} })
116852025-04-30T06:30:18.622ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
116862025-04-30T06:30:18.622ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116872025-04-30T06:30:18.633ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116882025-04-30T06:30:18.633ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116892025-04-30T06:30:18.633ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
116902025-04-30T06:30:18.634ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116912025-04-30T06:30:18.637ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116922025-04-30T06:30:18.637ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
116932025-04-30T06:30:18.637ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116942025-04-30T06:30:18.639ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa20c852-0758-45f2-a8e1-cd69116d27a6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
116952025-04-30T06:30:18.641ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
116962025-04-30T06:30:18.643ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
116972025-04-30T06:30:18.644ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
116982025-04-30T06:30:18.645ZTRACtest_image_promotion (dropshot): incoming request local_addr = 127.0.0.1:60931 method = GET remote_addr = 127.0.0.1:53785 req_id = 424e8f34-c086-4775-bc12-d8decf0275e8 uri = /d9b51ab0-9dce-44da-9764-8fd37f3b0498
116992025-04-30T06:30:18.649ZINFOtest_image_promotion (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 4679 local_addr = 127.0.0.1:60931 method = GET remote_addr = 127.0.0.1:53785 req_id = 424e8f34-c086-4775-bc12-d8decf0275e8 response_code = 200 uri = /d9b51ab0-9dce-44da-9764-8fd37f3b0498
117002025-04-30T06:30:18.650ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eb03d5b-976d-47d2-9272-150766550cd8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117012025-04-30T06:30:18.650ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
117022025-04-30T06:30:18.651ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
117032025-04-30T06:30:18.658ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
117042025-04-30T06:30:18.658ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
117052025-04-30T06:30:18.658ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
117062025-04-30T06:30:18.661ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b2bddad-33c0-44b2-9328-fd4283372559", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117072025-04-30T06:30:18.661ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
117082025-04-30T06:30:18.662ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f67749a-b868-4a16-bd38-6aeba2681095", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117092025-04-30T06:30:18.663ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
117102025-04-30T06:30:18.664ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c0e77bf-ebc1-497c-a531-9bb8fb74b5d8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117112025-04-30T06:30:18.664ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
117122025-04-30T06:30:18.665ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "945792b8-88dd-4119-8d01-8f8fee1b492f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117132025-04-30T06:30:18.669ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
117142025-04-30T06:30:18.670ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e71e364c-8dbc-47c8-ab8a-7e58b24c6979", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117152025-04-30T06:30:18.671ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
117162025-04-30T06:30:18.672ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ec0c397-4334-481d-ba9f-6c20d9437d1e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117172025-04-30T06:30:18.672ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
117182025-04-30T06:30:18.676ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f6def02-1ebb-4f57-b623-9dd4124a4819", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117192025-04-30T06:30:18.676ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
117202025-04-30T06:30:18.678ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1d8695c-95ee-41c6-9b33-454bff5b9fb7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117212025-04-30T06:30:18.678ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
117222025-04-30T06:30:18.680ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f55a17c-e63a-46b8-a05f-2d00c606a0a6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117232025-04-30T06:30:18.680ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
117242025-04-30T06:30:18.685ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2175d296-86a8-4d5f-8ea6-1c4e3851f2f0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117252025-04-30T06:30:18.686ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
117262025-04-30T06:30:18.687ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
117272025-04-30T06:30:18.689ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcb4ebb6-0be2-490d-8a87-5fb4e3993894", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117282025-04-30T06:30:18.689ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
117292025-04-30T06:30:18.690ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cfc75a1-d848-46b8-8de4-ac9947b262c7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117302025-04-30T06:30:18.691ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
117312025-04-30T06:30:18.691ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3284338e-8b5c-4189-9c2f-1612cb36e7f3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117322025-04-30T06:30:18.692ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
117332025-04-30T06:30:18.693ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f65af604-1196-435b-8da3-9e5d4d405a4b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117342025-04-30T06:30:18.693ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
117352025-04-30T06:30:18.694ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fda1a1c8-1a1a-45a7-99e8-f4ca2c9267a3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117362025-04-30T06:30:18.694ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
117372025-04-30T06:30:18.695ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d83e88b-c6e4-48ee-beaa-ccce0c1e02fa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117382025-04-30T06:30:18.696ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117392025-04-30T06:30:18.696ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117402025-04-30T06:30:18.696ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117412025-04-30T06:30:18.696ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117422025-04-30T06:30:18.696ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117432025-04-30T06:30:18.698ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117442025-04-30T06:30:18.706ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117452025-04-30T06:30:18.706ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117462025-04-30T06:30:18.706ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117472025-04-30T06:30:18.706ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117482025-04-30T06:30:18.706ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117492025-04-30T06:30:18.706ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117502025-04-30T06:30:18.709ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117512025-04-30T06:30:18.713ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117522025-04-30T06:30:18.714ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117532025-04-30T06:30:18.714ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117542025-04-30T06:30:18.714ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117552025-04-30T06:30:18.811ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
117562025-04-30T06:30:18.811ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
117572025-04-30T06:30:18.813ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117582025-04-30T06:30:18.813ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117592025-04-30T06:30:18.813ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117602025-04-30T06:30:18.813ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
117612025-04-30T06:30:18.816ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117622025-04-30T06:30:18.817ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:41257 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
117632025-04-30T06:30:18.817ZDEBGtest_image_promotion (results-sink): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117642025-04-30T06:30:18.817ZDEBGtest_image_promotion (results-sink-cluster): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117652025-04-30T06:30:18.818ZDEBGtest_image_promotion (clickhouse-client): unrolling 6 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
117662025-04-30T06:30:18.823ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
117672025-04-30T06:30:18.824ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:41257 req_id = 50664c69-1d68-45b5-9c00-3eeafdeee520 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
117682025-04-30T06:30:18.824ZINFOtest_image_promotion (dropshot): request completed latency_us = 675 local_addr = [::1]:60070 method = GET remote_addr = [::1]:41257 req_id = 50664c69-1d68-45b5-9c00-3eeafdeee520 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
117692025-04-30T06:30:18.825ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117702025-04-30T06:30:18.835ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
117712025-04-30T06:30:18.835ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117722025-04-30T06:30:18.835ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
117732025-04-30T06:30:18.836ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117742025-04-30T06:30:18.836ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "50664c69-1d68-45b5-9c00-3eeafdeee520", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
117752025-04-30T06:30:18.840ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
117762025-04-30T06:30:18.840ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
117772025-04-30T06:30:18.840ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
117782025-04-30T06:30:18.853ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
117792025-04-30T06:30:18.853ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117802025-04-30T06:30:18.853ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117812025-04-30T06:30:18.853ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117822025-04-30T06:30:18.857ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117832025-04-30T06:30:18.859ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 12 table_name = fields_string
117842025-04-30T06:30:18.859ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
117852025-04-30T06:30:18.862ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
117862025-04-30T06:30:18.866ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117872025-04-30T06:30:18.867ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
117882025-04-30T06:30:18.868ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117892025-04-30T06:30:18.869ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117902025-04-30T06:30:18.874ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
117912025-04-30T06:30:18.875ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117922025-04-30T06:30:18.875ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117932025-04-30T06:30:18.875ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117942025-04-30T06:30:18.902ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 5 table_name = fields_u16
117952025-04-30T06:30:18.902ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
117962025-04-30T06:30:18.903ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 85 a1 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63688 req_id = c4a123ec-cad6-4965-82d0-23766daf1eed
117972025-04-30T06:30:18.903ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34209,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63688 req_id = c4a123ec-cad6-4965-82d0-23766daf1eed
117982025-04-30T06:30:18.906ZDEBGtest_image_promotion (store): query key key = _dendrite._tcp
117992025-04-30T06:30:18.906ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118002025-04-30T06:30:18.907ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63688 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c4a123ec-cad6-4965-82d0-23766daf1eed
118012025-04-30T06:30:18.909ZDEBGtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3483, tv_nsec: 707329951 } })
118022025-04-30T06:30:18.909ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
118032025-04-30T06:30:18.910ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
118042025-04-30T06:30:18.918ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 5 table_name = fields_uuid
118052025-04-30T06:30:18.918ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
118062025-04-30T06:30:18.919ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 24 1f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e $...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49656 req_id = 5ece60f0-9f19-47b2-8339-415b4655d29f
118072025-04-30T06:30:18.919ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9247,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49656 req_id = 5ece60f0-9f19-47b2-8339-415b4655d29f
118082025-04-30T06:30:18.920ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118092025-04-30T06:30:18.920ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49656 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5ece60f0-9f19-47b2-8339-415b4655d29f
118102025-04-30T06:30:18.920ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 87 3a 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .:..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:49895 req_id = e1717453-1cc4-4f65-beb4-f3ee72473e7a
118112025-04-30T06:30:18.921ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34618,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49895 req_id = e1717453-1cc4-4f65-beb4-f3ee72473e7a
118122025-04-30T06:30:18.921ZDEBGtest_image_promotion (store): query key key = _dendrite._tcp
118132025-04-30T06:30:18.921ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118142025-04-30T06:30:18.921ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49895 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e1717453-1cc4-4f65-beb4-f3ee72473e7a
118152025-04-30T06:30:18.922ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
118162025-04-30T06:30:18.925ZINFOtest_image_promotion (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
118172025-04-30T06:30:19.029ZINFOtest_image_promotion (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
118182025-04-30T06:30:19.029ZDEBGtest_image_promotion (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
118192025-04-30T06:30:19.030ZDEBGtest_image_promotion (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
118202025-04-30T06:30:19.030ZWARNtest_image_promotion (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
118212025-04-30T06:30:19.033ZWARNtest_image_promotion (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
118222025-04-30T06:30:19.033ZDEBGtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3483, tv_nsec: 832605504 } })
118232025-04-30T06:30:19.033ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
118242025-04-30T06:30:19.038ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
118252025-04-30T06:30:19.041ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 5 table_name = measurements_histogramu64
118262025-04-30T06:30:19.041ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
118272025-04-30T06:30:19.041ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 4e 08 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e N...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59406 req_id = 83c9e52d-fbe7-4051-9c01-ddd28144dbe1
118282025-04-30T06:30:19.042ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19976,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59406 req_id = 83c9e52d-fbe7-4051-9c01-ddd28144dbe1
118292025-04-30T06:30:19.042ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118302025-04-30T06:30:19.042ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59406 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 83c9e52d-fbe7-4051-9c01-ddd28144dbe1
118312025-04-30T06:30:19.046ZINFOtest_image_promotion (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
118322025-04-30T06:30:19.147ZINFOtest_image_promotion (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
118332025-04-30T06:30:19.147ZDEBGtest_image_promotion (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
118342025-04-30T06:30:19.149ZDEBGtest_image_promotion (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
118352025-04-30T06:30:19.150ZWARNtest_image_promotion (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
118362025-04-30T06:30:19.151ZWARNtest_image_promotion (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
118372025-04-30T06:30:19.152ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 93 50 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .P..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52975 req_id = beeba014-a4c0-4a26-a256-d4b5c6f2b2a4
118382025-04-30T06:30:19.152ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37712,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52975 req_id = beeba014-a4c0-4a26-a256-d4b5c6f2b2a4
118392025-04-30T06:30:19.152ZDEBGtest_image_promotion (store): query key key = _dendrite._tcp
118402025-04-30T06:30:19.153ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118412025-04-30T06:30:19.153ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52975 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = beeba014-a4c0-4a26-a256-d4b5c6f2b2a4
118422025-04-30T06:30:19.154ZDEBGtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3483, tv_nsec: 953477995 } })
118432025-04-30T06:30:19.154ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
118442025-04-30T06:30:19.155ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
118452025-04-30T06:30:19.155ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
118462025-04-30T06:30:19.186ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 78 ec 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e x...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53332 req_id = 57203ccd-cae9-4eb2-a7b1-123d4979e9a5
118472025-04-30T06:30:19.187ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30956,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53332 req_id = 57203ccd-cae9-4eb2-a7b1-123d4979e9a5
118482025-04-30T06:30:19.187ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118492025-04-30T06:30:19.187ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53332 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 57203ccd-cae9-4eb2-a7b1-123d4979e9a5
118502025-04-30T06:30:19.190ZINFOtest_image_promotion (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
118512025-04-30T06:30:19.272ZINFOtest_image_promotion (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
118522025-04-30T06:30:19.272ZDEBGtest_image_promotion (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
118532025-04-30T06:30:19.273ZDEBGtest_image_promotion (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
118542025-04-30T06:30:19.275ZWARNtest_image_promotion (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
118552025-04-30T06:30:19.278ZWARNtest_image_promotion (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
118562025-04-30T06:30:19.299ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
118572025-04-30T06:30:19.300ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
118582025-04-30T06:30:19.300ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 863.848882ms iteration = 42 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
118592025-04-30T06:30:19.300ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 43 reason = Signaled
118602025-04-30T06:30:19.300ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
118612025-04-30T06:30:19.300ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118622025-04-30T06:30:19.300ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118632025-04-30T06:30:19.300ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118642025-04-30T06:30:19.301ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118652025-04-30T06:30:19.310ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118662025-04-30T06:30:19.316ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
118672025-04-30T06:30:19.316ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118682025-04-30T06:30:19.316ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118692025-04-30T06:30:19.316ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118702025-04-30T06:30:19.317ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
118712025-04-30T06:30:19.324ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
118722025-04-30T06:30:19.324ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
118732025-04-30T06:30:19.332ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: dc 82 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63754 req_id = 09e266e3-0ac2-4dd4-944d-88791b29435f
118742025-04-30T06:30:19.332ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56450,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63754 req_id = 09e266e3-0ac2-4dd4-944d-88791b29435f
118752025-04-30T06:30:19.332ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
118762025-04-30T06:30:19.333ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118772025-04-30T06:30:19.333ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63754 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 09e266e3-0ac2-4dd4-944d-88791b29435f
118782025-04-30T06:30:19.334ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3484, tv_nsec: 133004269 } })
118792025-04-30T06:30:19.334ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
118802025-04-30T06:30:19.337ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 06 67 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .g..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61322 req_id = d51381ba-5280-470f-9436-aa46b07a4258
118812025-04-30T06:30:19.337ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1639,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61322 req_id = d51381ba-5280-470f-9436-aa46b07a4258
118822025-04-30T06:30:19.338ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118832025-04-30T06:30:19.338ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61322 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d51381ba-5280-470f-9436-aa46b07a4258
118842025-04-30T06:30:19.394ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
118852025-04-30T06:30:19.395ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118862025-04-30T06:30:19.395ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118872025-04-30T06:30:19.395ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118882025-04-30T06:30:19.395ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
118892025-04-30T06:30:19.396ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118902025-04-30T06:30:19.399ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ba c3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52440 req_id = a5991cf6-2d9d-4c11-abe0-1f1152af8f02
118912025-04-30T06:30:19.399ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47811,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52440 req_id = a5991cf6-2d9d-4c11-abe0-1f1152af8f02
118922025-04-30T06:30:19.399ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
118932025-04-30T06:30:19.400ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47811, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52440 req_id = a5991cf6-2d9d-4c11-abe0-1f1152af8f02
118942025-04-30T06:30:19.400ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
118952025-04-30T06:30:19.400ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
118962025-04-30T06:30:19.400ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
118972025-04-30T06:30:19.401ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118982025-04-30T06:30:19.410ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118992025-04-30T06:30:19.410ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119002025-04-30T06:30:19.410ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
119012025-04-30T06:30:19.410ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119022025-04-30T06:30:19.427ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119032025-04-30T06:30:19.427ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
119042025-04-30T06:30:19.428ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119052025-04-30T06:30:19.429ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4fe1541-8792-46cc-9de7-b68f35557430", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119062025-04-30T06:30:19.429ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
119072025-04-30T06:30:19.447ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79c66d77-2bb2-4642-be27-90f95e2b6770", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119082025-04-30T06:30:19.447ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
119092025-04-30T06:30:19.451ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "495a4bbf-dda3-406d-869a-208fbc3298df", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119102025-04-30T06:30:19.451ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
119112025-04-30T06:30:19.454ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e2770de-5ddf-4a90-b61a-739ce76ce70f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119122025-04-30T06:30:19.454ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
119132025-04-30T06:30:19.458ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d95bb34b-279d-4f14-a7b0-f85da0dbe600", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119142025-04-30T06:30:19.458ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
119152025-04-30T06:30:19.461ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1213a04-ca99-4019-bc61-70dc329537ac", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119162025-04-30T06:30:19.461ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
119172025-04-30T06:30:19.465ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "478f6108-13ba-43af-8543-e4f11bc1de1d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119182025-04-30T06:30:19.465ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
119192025-04-30T06:30:19.469ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "857c9ff7-15cc-4ce4-8984-e516e7b31792", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119202025-04-30T06:30:19.469ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
119212025-04-30T06:30:19.470ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dba0a26-ba22-4143-b9cd-0c2c0bf25301", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119222025-04-30T06:30:19.470ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
119232025-04-30T06:30:19.474ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
119242025-04-30T06:30:19.475ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "49a47401-9919-4515-a391-790bed96daed", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119252025-04-30T06:30:19.475ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
119262025-04-30T06:30:19.476ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "585de8ca-024e-4bf3-9460-bf40b6c5c225", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119272025-04-30T06:30:19.476ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
119282025-04-30T06:30:19.478ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4a14d60-658b-4dc4-827e-88087bdf8590", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119292025-04-30T06:30:19.478ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
119302025-04-30T06:30:19.479ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02e2b252-5195-47c5-8d13-e7973eefd32d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119312025-04-30T06:30:19.479ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
119322025-04-30T06:30:19.481ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "902b0c7f-28fa-451e-b614-a827789cf2b9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119332025-04-30T06:30:19.481ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
119342025-04-30T06:30:19.485ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37ef8719-5e5a-41c7-b5fb-5132d2bd0486", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119352025-04-30T06:30:19.485ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
119362025-04-30T06:30:19.489ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2173e0f-a8e0-46e8-b871-f25f7ffe9fd9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119372025-04-30T06:30:19.489ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
119382025-04-30T06:30:19.491ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c86b3031-4f42-45ba-b211-fbb73c90d508", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119392025-04-30T06:30:19.493ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
119402025-04-30T06:30:19.497ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cdafd96-a7b8-45cf-a56f-466bf002d13b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119412025-04-30T06:30:19.497ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
119422025-04-30T06:30:19.502ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b36e5c58-ff63-4d75-b1dc-9f3c85856fcf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119432025-04-30T06:30:19.503ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
119442025-04-30T06:30:19.506ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3744a3a8-6d17-4580-97f2-e9c8ac158be0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119452025-04-30T06:30:19.506ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
119462025-04-30T06:30:19.507ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31111250-7050-4a46-be6a-88ba19663181", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119472025-04-30T06:30:19.508ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
119482025-04-30T06:30:19.509ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8b4ae25-52f4-44dd-951f-c23c89dbb336", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119492025-04-30T06:30:19.510ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
119502025-04-30T06:30:19.511ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10301d59-3b9b-43b5-a965-85eff75d6e02", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119512025-04-30T06:30:19.512ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
119522025-04-30T06:30:19.516ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89568357-d969-4fa4-9e42-b7e4282eab5e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119532025-04-30T06:30:19.516ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
119542025-04-30T06:30:19.520ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dc079f3-90be-4c4e-9ac5-dd9f09b89574", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119552025-04-30T06:30:19.520ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
119562025-04-30T06:30:19.523ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "642e35eb-c56d-4f50-8157-f47d3a97266c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119572025-04-30T06:30:19.523ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
119582025-04-30T06:30:19.525ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119592025-04-30T06:30:19.525ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119602025-04-30T06:30:19.525ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119612025-04-30T06:30:19.525ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
119622025-04-30T06:30:19.526ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119632025-04-30T06:30:19.527ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c0312cd-1579-48b1-8583-474427c9e424", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119642025-04-30T06:30:19.527ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
119652025-04-30T06:30:19.529ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119662025-04-30T06:30:19.538ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119672025-04-30T06:30:19.538ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119682025-04-30T06:30:19.538ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
119692025-04-30T06:30:19.538ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119702025-04-30T06:30:19.541ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119712025-04-30T06:30:19.541ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119722025-04-30T06:30:19.542ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119732025-04-30T06:30:19.542ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
119742025-04-30T06:30:19.544ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119752025-04-30T06:30:19.547ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119762025-04-30T06:30:19.549ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119772025-04-30T06:30:19.549ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119782025-04-30T06:30:19.549ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119792025-04-30T06:30:19.551ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7c2d9b0-33b5-4b31-b13f-1ee96e78d261", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119802025-04-30T06:30:19.551ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
119812025-04-30T06:30:19.554ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d752e7ab-d9f1-4925-a651-46c917a1e418", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119822025-04-30T06:30:19.554ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
119832025-04-30T06:30:19.559ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16e54378-49ca-41b1-97cb-7bda349a2a7c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119842025-04-30T06:30:19.559ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
119852025-04-30T06:30:19.561ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eed34644-379c-422a-9b6c-c5e4653414f0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119862025-04-30T06:30:19.561ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
119872025-04-30T06:30:19.567ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfc5843d-7f29-4c2e-8e2f-64d4ace97792", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119882025-04-30T06:30:19.568ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
119892025-04-30T06:30:19.571ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "466b68cf-9090-40e2-9946-76de3ac4b169", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119902025-04-30T06:30:19.573ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
119912025-04-30T06:30:19.576ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c38957c5-b812-4dde-ac51-8169125cfb80", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119922025-04-30T06:30:19.577ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
119932025-04-30T06:30:19.578ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c290d28d-a88b-42c0-b63b-db9e8b2a87b7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119942025-04-30T06:30:19.581ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
119952025-04-30T06:30:19.586ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
119962025-04-30T06:30:19.588ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54c4213a-f72b-4ee3-b854-912a75b0b9f4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119972025-04-30T06:30:19.588ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
119982025-04-30T06:30:19.591ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "505e10e7-1975-4a25-b196-802213b55016", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
119992025-04-30T06:30:19.592ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
120002025-04-30T06:30:19.594ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7fdcf82-9c24-4b87-9aff-a7374a38203c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
120012025-04-30T06:30:19.594ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
120022025-04-30T06:30:19.604ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee96e5c9-e4b0-4c5a-bbfe-abb4feaa074d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
120032025-04-30T06:30:19.604ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
120042025-04-30T06:30:19.608ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58c7e630-e0d2-46dd-b14b-91f8033eef1c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
120052025-04-30T06:30:19.608ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
120062025-04-30T06:30:19.611ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86890fbf-6b9e-4fe7-90b5-4e22a0f47f4a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
120072025-04-30T06:30:19.611ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
120082025-04-30T06:30:19.612ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d1f5d51-4d34-493e-af11-1ef93f6c98d0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
120092025-04-30T06:30:19.613ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
120102025-04-30T06:30:19.615ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ae4d6ef-a63a-4449-8212-c4977b0db550", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
120112025-04-30T06:30:19.615ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
120122025-04-30T06:30:19.616ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f39c5fd-0845-4e38-ba57-71bc64a3ec48", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
120132025-04-30T06:30:19.617ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
120142025-04-30T06:30:19.619ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60cd2461-4d8d-4632-8fff-8e5b662562e0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:18 GMT"} })
120152025-04-30T06:30:19.619ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
120162025-04-30T06:30:19.619ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120172025-04-30T06:30:19.619ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120182025-04-30T06:30:19.620ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120192025-04-30T06:30:19.622ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120202025-04-30T06:30:19.626ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120212025-04-30T06:30:19.631ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
120222025-04-30T06:30:19.631ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
120232025-04-30T06:30:19.631ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120242025-04-30T06:30:19.632ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120252025-04-30T06:30:19.632ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120262025-04-30T06:30:19.632ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120272025-04-30T06:30:19.634ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120282025-04-30T06:30:19.638ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
120292025-04-30T06:30:19.638ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120302025-04-30T06:30:19.638ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120312025-04-30T06:30:19.639ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120322025-04-30T06:30:19.639ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
120332025-04-30T06:30:19.639ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120342025-04-30T06:30:19.639ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120352025-04-30T06:30:19.639ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
120362025-04-30T06:30:19.640ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120372025-04-30T06:30:19.702ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
120382025-04-30T06:30:19.702ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
120392025-04-30T06:30:19.703ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120402025-04-30T06:30:19.708ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
120412025-04-30T06:30:19.708ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120422025-04-30T06:30:19.708ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
120432025-04-30T06:30:19.709ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120442025-04-30T06:30:19.709ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:39531 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
120452025-04-30T06:30:19.709ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:39531 req_id = c7fe0623-c9ff-45f9-bb96-692be90aa300 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
120462025-04-30T06:30:19.710ZINFOtest_image_promotion (dropshot): request completed latency_us = 581 local_addr = [::1]:60070 method = GET remote_addr = [::1]:39531 req_id = c7fe0623-c9ff-45f9-bb96-692be90aa300 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
120472025-04-30T06:30:19.710ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7fe0623-c9ff-45f9-bb96-692be90aa300", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
120482025-04-30T06:30:19.712ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
120492025-04-30T06:30:19.712ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
120502025-04-30T06:30:19.713ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
120512025-04-30T06:30:19.715ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
120522025-04-30T06:30:19.716ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120532025-04-30T06:30:19.716ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120542025-04-30T06:30:19.716ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120552025-04-30T06:30:19.717ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120562025-04-30T06:30:19.718ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120572025-04-30T06:30:19.718ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
120582025-04-30T06:30:19.718ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120592025-04-30T06:30:19.720ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120602025-04-30T06:30:19.725ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
120612025-04-30T06:30:19.725ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120622025-04-30T06:30:19.725ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120632025-04-30T06:30:19.726ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120642025-04-30T06:30:19.750ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
120652025-04-30T06:30:19.751ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
120662025-04-30T06:30:19.751ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
120672025-04-30T06:30:19.753ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
120682025-04-30T06:30:19.761ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
120692025-04-30T06:30:19.763ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
120702025-04-30T06:30:19.764ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
120712025-04-30T06:30:19.766ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
120722025-04-30T06:30:19.767ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
120732025-04-30T06:30:19.767ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 466.860048ms iteration = 43 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
120742025-04-30T06:30:19.767ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 44 reason = Signaled
120752025-04-30T06:30:19.767ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
120762025-04-30T06:30:19.767ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120772025-04-30T06:30:19.767ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120782025-04-30T06:30:19.767ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120792025-04-30T06:30:19.770ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120802025-04-30T06:30:19.771ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
120812025-04-30T06:30:19.771ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
120822025-04-30T06:30:19.774ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120832025-04-30T06:30:19.780ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
120842025-04-30T06:30:19.780ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120852025-04-30T06:30:19.780ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120862025-04-30T06:30:19.781ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120872025-04-30T06:30:19.784ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
120882025-04-30T06:30:19.784ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
120892025-04-30T06:30:19.785ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 23 24 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 #$..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58188 req_id = fb494267-7630-4a96-80d7-6677e34ecf05
120902025-04-30T06:30:19.785ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8996,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58188 req_id = fb494267-7630-4a96-80d7-6677e34ecf05
120912025-04-30T06:30:19.786ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
120922025-04-30T06:30:19.786ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120932025-04-30T06:30:19.786ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58188 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = fb494267-7630-4a96-80d7-6677e34ecf05
120942025-04-30T06:30:19.786ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3484, tv_nsec: 585632862 } })
120952025-04-30T06:30:19.786ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
120962025-04-30T06:30:19.787ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e6 33 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .3..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65152 req_id = e7042965-5458-4fe8-b569-e0ed27207866
120972025-04-30T06:30:19.787ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58931,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65152 req_id = e7042965-5458-4fe8-b569-e0ed27207866
120982025-04-30T06:30:19.787ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120992025-04-30T06:30:19.787ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65152 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e7042965-5458-4fe8-b569-e0ed27207866
121002025-04-30T06:30:19.854ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
121012025-04-30T06:30:19.855ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121022025-04-30T06:30:19.855ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121032025-04-30T06:30:19.855ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121042025-04-30T06:30:19.855ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
121052025-04-30T06:30:19.856ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121062025-04-30T06:30:19.856ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
121072025-04-30T06:30:19.859ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 24 0b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c $............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65286 req_id = 61f01d54-14e3-4798-aced-168f886f4428
121082025-04-30T06:30:19.859ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9227,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65286 req_id = 61f01d54-14e3-4798-aced-168f886f4428
121092025-04-30T06:30:19.860ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
121102025-04-30T06:30:19.860ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9227, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65286 req_id = 61f01d54-14e3-4798-aced-168f886f4428
121112025-04-30T06:30:19.861ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
121122025-04-30T06:30:19.861ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
121132025-04-30T06:30:19.861ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
121142025-04-30T06:30:19.862ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121152025-04-30T06:30:19.868ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121162025-04-30T06:30:19.868ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121172025-04-30T06:30:19.868ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
121182025-04-30T06:30:19.869ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121192025-04-30T06:30:19.877ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ed6a282-b745-462e-9216-ec5eb0cb0a4b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121202025-04-30T06:30:19.877ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
121212025-04-30T06:30:19.881ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121222025-04-30T06:30:19.881ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
121232025-04-30T06:30:19.883ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121242025-04-30T06:30:19.886ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5728d0e4-1a1f-4e13-bc29-d5e123dd7836", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121252025-04-30T06:30:19.886ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
121262025-04-30T06:30:19.890ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4ffc454-dba7-4453-be06-0bb61b65c03e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121272025-04-30T06:30:19.890ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
121282025-04-30T06:30:19.899ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f9f72d5-15ca-4f62-854c-ebe937e6def2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121292025-04-30T06:30:19.900ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
121302025-04-30T06:30:19.909ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be4bea8d-8e4e-49db-b52d-c7dca7d4c538", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121312025-04-30T06:30:19.909ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
121322025-04-30T06:30:19.918ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
121332025-04-30T06:30:19.923ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0f5a8cd-4954-4e64-85b8-cfc7ca905b1a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121342025-04-30T06:30:19.923ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
121352025-04-30T06:30:19.931ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b98e2b7f-e3b4-474f-af7e-a074c1c1541f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121362025-04-30T06:30:19.931ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
121372025-04-30T06:30:19.938ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04b134c8-51a6-44ae-9da2-797e9009a4ae", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121382025-04-30T06:30:19.939ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
121392025-04-30T06:30:19.944ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "235852bf-9628-4384-8067-187663a189f4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121402025-04-30T06:30:19.944ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
121412025-04-30T06:30:19.946ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7b69920-545b-4559-b87a-f40291057e21", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121422025-04-30T06:30:19.948ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
121432025-04-30T06:30:19.970ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c774d120-67f9-4163-bbb7-3dd2f84d08db", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121442025-04-30T06:30:19.970ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
121452025-04-30T06:30:19.970ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121462025-04-30T06:30:19.971ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121472025-04-30T06:30:19.971ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121482025-04-30T06:30:19.971ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
121492025-04-30T06:30:19.972ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121502025-04-30T06:30:20.002ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121512025-04-30T06:30:20.017ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121522025-04-30T06:30:20.017ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121532025-04-30T06:30:20.018ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
121542025-04-30T06:30:20.021ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121552025-04-30T06:30:20.025ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4cda72e-023a-4423-bd52-727e876f3bc9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121562025-04-30T06:30:20.026ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
121572025-04-30T06:30:20.038ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121582025-04-30T06:30:20.038ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
121592025-04-30T06:30:20.039ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121602025-04-30T06:30:20.066ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d7786b3-654d-4a82-881f-0ddd39bfb099", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121612025-04-30T06:30:20.066ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
121622025-04-30T06:30:20.074ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db32010f-6430-4c78-a07d-d293ae20f4eb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121632025-04-30T06:30:20.074ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
121642025-04-30T06:30:20.089ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "193bdb8d-8795-41e4-bc75-3cc7b6af8315", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121652025-04-30T06:30:20.089ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
121662025-04-30T06:30:20.099ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f22c494-e179-4f5f-82b8-49e4dcb88d05", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121672025-04-30T06:30:20.099ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
121682025-04-30T06:30:20.105ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "acb901fa-7c69-46ec-9231-4485489a7cbb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121692025-04-30T06:30:20.105ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
121702025-04-30T06:30:20.110ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
121712025-04-30T06:30:20.112ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f00830a2-83f0-4d1b-ab03-5763d2efac0b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121722025-04-30T06:30:20.112ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
121732025-04-30T06:30:20.115ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c5efa39-fcd6-49cd-a26a-1bae867637af", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121742025-04-30T06:30:20.115ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
121752025-04-30T06:30:20.116ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8e80f19-c59d-4a61-b818-28ca7b17581e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121762025-04-30T06:30:20.119ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
121772025-04-30T06:30:20.121ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e5d5927-3ba3-418a-8d55-0d1e06ca1fb7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121782025-04-30T06:30:20.122ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
121792025-04-30T06:30:20.124ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "033fb22a-fd9c-4a64-8841-3175a513d6dd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121802025-04-30T06:30:20.124ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
121812025-04-30T06:30:20.125ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9da60cf-9a4c-4b0c-b5e4-2b523946c934", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121822025-04-30T06:30:20.125ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
121832025-04-30T06:30:20.126ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6afc77b9-d319-4ed7-8829-89ba87a2ca45", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121842025-04-30T06:30:20.127ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
121852025-04-30T06:30:20.129ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d549ff6-54ae-4e41-9481-2ca1d2bafb37", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121862025-04-30T06:30:20.129ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
121872025-04-30T06:30:20.132ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb93a189-c867-467a-aec8-d475cab6e564", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121882025-04-30T06:30:20.133ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
121892025-04-30T06:30:20.136ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de554c13-203b-49b9-b0c9-e8fff4e0616e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121902025-04-30T06:30:20.136ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
121912025-04-30T06:30:20.142ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca5f1c6a-6d99-4304-b65d-90881d48cccc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121922025-04-30T06:30:20.142ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
121932025-04-30T06:30:20.146ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "523d8959-17fd-43cf-873c-f940e00d0889", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121942025-04-30T06:30:20.146ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
121952025-04-30T06:30:20.150ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ecdbc11-d9e6-4547-a174-a31a0eb4fa85", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121962025-04-30T06:30:20.150ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
121972025-04-30T06:30:20.151ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "289bcf97-717c-418f-858a-3dff09512b70", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
121982025-04-30T06:30:20.151ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
121992025-04-30T06:30:20.153ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8e37700-ad35-46b5-9be5-394a70375019", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122002025-04-30T06:30:20.153ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
122012025-04-30T06:30:20.154ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd437752-8c4a-44c6-b874-9dab70e02db1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122022025-04-30T06:30:20.155ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
122032025-04-30T06:30:20.156ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c189d27-752d-4d51-91c8-f2db5905ca89", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122042025-04-30T06:30:20.156ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
122052025-04-30T06:30:20.158ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d83286f-f2b3-43f8-a821-2c138113eb19", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122062025-04-30T06:30:20.159ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
122072025-04-30T06:30:20.161ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122082025-04-30T06:30:20.161ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122092025-04-30T06:30:20.161ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122102025-04-30T06:30:20.161ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
122112025-04-30T06:30:20.162ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122122025-04-30T06:30:20.164ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "53ec3842-5f97-4446-b6a4-6d757d858049", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122132025-04-30T06:30:20.164ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
122142025-04-30T06:30:20.164ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122152025-04-30T06:30:20.170ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122162025-04-30T06:30:20.170ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122172025-04-30T06:30:20.170ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
122182025-04-30T06:30:20.171ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122192025-04-30T06:30:20.174ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122202025-04-30T06:30:20.174ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
122212025-04-30T06:30:20.178ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122222025-04-30T06:30:20.188ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58832ee4-cc0b-46b8-972d-540c547901d2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122232025-04-30T06:30:20.188ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
122242025-04-30T06:30:20.191ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bc92e98-6342-4bf8-a707-ddc294a0eea2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122252025-04-30T06:30:20.191ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
122262025-04-30T06:30:20.194ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a088175-0009-4931-b93b-b7be5c6ce193", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122272025-04-30T06:30:20.195ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
122282025-04-30T06:30:20.197ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61668133-7a4b-44ca-be77-a0eb5024ee83", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122292025-04-30T06:30:20.197ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
122302025-04-30T06:30:20.199ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2189489e-51eb-49ff-b65b-cf13a1cb0968", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122312025-04-30T06:30:20.199ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
122322025-04-30T06:30:20.200ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "862648a4-85eb-47eb-9c32-9ee2e5fb170b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122332025-04-30T06:30:20.200ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
122342025-04-30T06:30:20.202ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a751231-29ba-499f-b3d6-07b7c048237b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122352025-04-30T06:30:20.203ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
122362025-04-30T06:30:20.205ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "177ecde8-7753-4403-aac2-1b0a8643baa1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122372025-04-30T06:30:20.206ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
122382025-04-30T06:30:20.207ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08b54d7e-1417-449a-b269-9e32b46d1617", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122392025-04-30T06:30:20.207ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
122402025-04-30T06:30:20.207ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122412025-04-30T06:30:20.207ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122422025-04-30T06:30:20.207ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122432025-04-30T06:30:20.208ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122442025-04-30T06:30:20.210ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122452025-04-30T06:30:20.218ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122462025-04-30T06:30:20.218ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
122472025-04-30T06:30:20.218ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122482025-04-30T06:30:20.218ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122492025-04-30T06:30:20.218ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122502025-04-30T06:30:20.219ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122512025-04-30T06:30:20.225ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122522025-04-30T06:30:20.228ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122532025-04-30T06:30:20.228ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122542025-04-30T06:30:20.228ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122552025-04-30T06:30:20.229ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122562025-04-30T06:30:20.229ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
122572025-04-30T06:30:20.315ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
122582025-04-30T06:30:20.315ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
122592025-04-30T06:30:20.316ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122602025-04-30T06:30:20.316ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122612025-04-30T06:30:20.316ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122622025-04-30T06:30:20.316ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
122632025-04-30T06:30:20.317ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122642025-04-30T06:30:20.317ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:45283 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
122652025-04-30T06:30:20.318ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:45283 req_id = 57ca55b8-d7e2-47f5-bda8-b2f062013674 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122662025-04-30T06:30:20.320ZINFOtest_image_promotion (dropshot): request completed latency_us = 2641 local_addr = [::1]:60070 method = GET remote_addr = [::1]:45283 req_id = 57ca55b8-d7e2-47f5-bda8-b2f062013674 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122672025-04-30T06:30:20.321ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "57ca55b8-d7e2-47f5-bda8-b2f062013674", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
122682025-04-30T06:30:20.323ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122692025-04-30T06:30:20.323ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122702025-04-30T06:30:20.323ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
122712025-04-30T06:30:20.327ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122722025-04-30T06:30:20.327ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122732025-04-30T06:30:20.327ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122742025-04-30T06:30:20.327ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122752025-04-30T06:30:20.328ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122762025-04-30T06:30:20.330ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122772025-04-30T06:30:20.337ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122782025-04-30T06:30:20.337ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122792025-04-30T06:30:20.337ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
122802025-04-30T06:30:20.337ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122812025-04-30T06:30:20.339ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122822025-04-30T06:30:20.344ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122832025-04-30T06:30:20.345ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122842025-04-30T06:30:20.345ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122852025-04-30T06:30:20.346ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122862025-04-30T06:30:20.347ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122872025-04-30T06:30:20.348ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
122882025-04-30T06:30:20.349ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122892025-04-30T06:30:20.376ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
122902025-04-30T06:30:20.377ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
122912025-04-30T06:30:20.377ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 610.416324ms iteration = 44 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
122922025-04-30T06:30:20.377ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 45 reason = Signaled
122932025-04-30T06:30:20.377ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122942025-04-30T06:30:20.377ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122952025-04-30T06:30:20.378ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122962025-04-30T06:30:20.378ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122972025-04-30T06:30:20.380ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122982025-04-30T06:30:20.387ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122992025-04-30T06:30:20.400ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
123002025-04-30T06:30:20.400ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123012025-04-30T06:30:20.400ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123022025-04-30T06:30:20.401ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123032025-04-30T06:30:20.402ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
123042025-04-30T06:30:20.405ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
123052025-04-30T06:30:20.405ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
123062025-04-30T06:30:20.406ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c5 8d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62245 req_id = 5d3a7943-ad67-426c-a088-843852190c07
123072025-04-30T06:30:20.406ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50573,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62245 req_id = 5d3a7943-ad67-426c-a088-843852190c07
123082025-04-30T06:30:20.406ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
123092025-04-30T06:30:20.407ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123102025-04-30T06:30:20.407ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62245 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5d3a7943-ad67-426c-a088-843852190c07
123112025-04-30T06:30:20.407ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3485, tv_nsec: 206548666 } })
123122025-04-30T06:30:20.407ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
123132025-04-30T06:30:20.408ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 06 99 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55489 req_id = 6b9fa37d-3969-43d1-b51b-a0c6b3a348d7
123142025-04-30T06:30:20.408ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1689,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55489 req_id = 6b9fa37d-3969-43d1-b51b-a0c6b3a348d7
123152025-04-30T06:30:20.408ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123162025-04-30T06:30:20.408ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55489 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6b9fa37d-3969-43d1-b51b-a0c6b3a348d7
123172025-04-30T06:30:20.480ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
123182025-04-30T06:30:20.480ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
123192025-04-30T06:30:20.480ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123202025-04-30T06:30:20.480ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123212025-04-30T06:30:20.480ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
123222025-04-30T06:30:20.481ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123232025-04-30T06:30:20.482ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b7 df 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51360 req_id = a04db5b3-7225-4a9d-8bb5-90e97acde66a
123242025-04-30T06:30:20.482ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47071,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51360 req_id = a04db5b3-7225-4a9d-8bb5-90e97acde66a
123252025-04-30T06:30:20.482ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
123262025-04-30T06:30:20.482ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47071, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51360 req_id = a04db5b3-7225-4a9d-8bb5-90e97acde66a
123272025-04-30T06:30:20.482ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
123282025-04-30T06:30:20.482ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
123292025-04-30T06:30:20.482ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
123302025-04-30T06:30:20.485ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123312025-04-30T06:30:20.490ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123322025-04-30T06:30:20.490ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123332025-04-30T06:30:20.490ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
123342025-04-30T06:30:20.490ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123352025-04-30T06:30:20.492ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6981769-f8d2-48d2-8af4-baf75b175914", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123362025-04-30T06:30:20.492ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
123372025-04-30T06:30:20.493ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123382025-04-30T06:30:20.493ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
123392025-04-30T06:30:20.494ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123402025-04-30T06:30:20.495ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef34be06-2baa-4764-972f-b2cab0447f0e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123412025-04-30T06:30:20.496ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
123422025-04-30T06:30:20.499ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae44b881-9b07-4bad-b598-11f9fffc844e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123432025-04-30T06:30:20.499ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
123442025-04-30T06:30:20.501ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d04b69d-8c1f-4905-9c46-5c148554e4f8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123452025-04-30T06:30:20.501ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
123462025-04-30T06:30:20.505ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
123472025-04-30T06:30:20.505ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30e53a92-4f07-42d7-92aa-ee470e69eb36", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123482025-04-30T06:30:20.505ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
123492025-04-30T06:30:20.506ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
123502025-04-30T06:30:20.506ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = c744d3cc-066e-4e95-a2f3-f02a87ae3fb8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
123512025-04-30T06:30:20.507ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = c744d3cc-066e-4e95-a2f3-f02a87ae3fb8 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
123522025-04-30T06:30:20.507ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = c744d3cc-066e-4e95-a2f3-f02a87ae3fb8 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
123532025-04-30T06:30:20.508ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = c744d3cc-066e-4e95-a2f3-f02a87ae3fb8 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
123542025-04-30T06:30:20.510ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 3985 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = c744d3cc-066e-4e95-a2f3-f02a87ae3fb8 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
123552025-04-30T06:30:20.511ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc6642b8-66cd-4884-8fad-549c6d7d7261", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123562025-04-30T06:30:20.511ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
123572025-04-30T06:30:20.511ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "c744d3cc-066e-4e95-a2f3-f02a87ae3fb8", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123582025-04-30T06:30:20.511ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
123592025-04-30T06:30:20.512ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 10679864-5661-4a4a-9d5d-02a197301121 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
123602025-04-30T06:30:20.512ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 10679864-5661-4a4a-9d5d-02a197301121 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
123612025-04-30T06:30:20.512ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 10679864-5661-4a4a-9d5d-02a197301121 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
123622025-04-30T06:30:20.513ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 10679864-5661-4a4a-9d5d-02a197301121 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
123632025-04-30T06:30:20.516ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94ecdee0-29ec-4151-a360-b4f509e8739a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123642025-04-30T06:30:20.516ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
123652025-04-30T06:30:20.516ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 4770 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 10679864-5661-4a4a-9d5d-02a197301121 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
123662025-04-30T06:30:20.518ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "10679864-5661-4a4a-9d5d-02a197301121", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123672025-04-30T06:30:20.518ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123682025-04-30T06:30:20.518ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
123692025-04-30T06:30:20.518ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
123702025-04-30T06:30:20.520ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df146072-7542-4381-8853-f6ee983652fc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123712025-04-30T06:30:20.520ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
123722025-04-30T06:30:20.533ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
123732025-04-30T06:30:20.533ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53f500b7-f031-4e05-934b-7af5ed52f1f3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123742025-04-30T06:30:20.533ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
123752025-04-30T06:30:20.536ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18ecf8ab-382d-4e06-a0fe-885549741b7b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123762025-04-30T06:30:20.536ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
123772025-04-30T06:30:20.537ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4d1a139-3df7-44cf-8172-3e51056c84c7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123782025-04-30T06:30:20.538ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
123792025-04-30T06:30:20.538ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123802025-04-30T06:30:20.538ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123812025-04-30T06:30:20.540ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123822025-04-30T06:30:20.542ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123832025-04-30T06:30:20.542ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123842025-04-30T06:30:20.542ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123852025-04-30T06:30:20.543ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f197854f-f960-42f7-bd59-e9b2ad847f34", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123862025-04-30T06:30:20.543ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
123872025-04-30T06:30:20.548ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d46e4e5-ac12-4173-a0fa-de6171832189", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123882025-04-30T06:30:20.548ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
123892025-04-30T06:30:20.552ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15a419fd-4614-4040-a9d2-7e194be55898", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123902025-04-30T06:30:20.552ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
123912025-04-30T06:30:20.556ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5acfae01-2c51-4e69-abaa-d571b932223b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123922025-04-30T06:30:20.556ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
123932025-04-30T06:30:20.560ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38cea06a-3be5-4912-aeb3-d040266f707b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123942025-04-30T06:30:20.560ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
123952025-04-30T06:30:20.563ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b81ff92-4439-43b0-8de6-a2770bc2eb3c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123962025-04-30T06:30:20.564ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
123972025-04-30T06:30:20.566ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d324299d-8720-4343-a840-1faef903c1be", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
123982025-04-30T06:30:20.566ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
123992025-04-30T06:30:20.570ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e0ed8f1-1065-4cca-aafc-4a9e7d941494", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124002025-04-30T06:30:20.570ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
124012025-04-30T06:30:20.573ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "99fe19fd-6e27-4c9b-8da1-e954a3534ade", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124022025-04-30T06:30:20.573ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
124032025-04-30T06:30:20.574ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14111881-6b9f-4a4e-9e7a-7e7736518225", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124042025-04-30T06:30:20.574ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
124052025-04-30T06:30:20.575ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fc432ec-566b-47a2-a3df-666f6bcd572f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124062025-04-30T06:30:20.575ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
124072025-04-30T06:30:20.577ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a41da623-a71c-4698-884c-192d71cc85b0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124082025-04-30T06:30:20.577ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
124092025-04-30T06:30:20.578ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bc46173-3458-4eb2-b5b4-a3bfeba4d62e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124102025-04-30T06:30:20.578ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
124112025-04-30T06:30:20.580ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04d638bd-0d27-4b5b-b1c2-3752e215a5e0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124122025-04-30T06:30:20.580ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
124132025-04-30T06:30:20.584ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "15e21a97-cd2d-4db8-8fb6-4f899af1621b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124142025-04-30T06:30:20.585ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
124152025-04-30T06:30:20.585ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
124162025-04-30T06:30:20.586ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124172025-04-30T06:30:20.586ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124182025-04-30T06:30:20.586ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
124192025-04-30T06:30:20.586ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124202025-04-30T06:30:20.589ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcaf9196-a535-43fd-b6b1-09dd4162951f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124212025-04-30T06:30:20.598ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
124222025-04-30T06:30:20.602ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124232025-04-30T06:30:20.608ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
124242025-04-30T06:30:20.608ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124252025-04-30T06:30:20.608ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
124262025-04-30T06:30:20.608ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124272025-04-30T06:30:20.612ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124282025-04-30T06:30:20.612ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
124292025-04-30T06:30:20.613ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124302025-04-30T06:30:20.613ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58cc96ff-c301-406c-a0f5-46996527b116", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124312025-04-30T06:30:20.613ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
124322025-04-30T06:30:20.616ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e473fe1-bc5f-4f52-8c01-be31103e8a49", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124332025-04-30T06:30:20.617ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
124342025-04-30T06:30:20.618ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca87286f-7f6c-41e2-b0e5-086c380387e3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124352025-04-30T06:30:20.618ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
124362025-04-30T06:30:20.619ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d87b2e07-e440-487e-8a1c-eadb81c4d69c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124372025-04-30T06:30:20.622ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
124382025-04-30T06:30:20.623ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db94f879-28b8-4961-b6bf-8b2df96ee012", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124392025-04-30T06:30:20.623ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
124402025-04-30T06:30:20.623ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b52858b-d92f-4c6b-a0d2-ec9890cea5a9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124412025-04-30T06:30:20.623ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
124422025-04-30T06:30:20.623ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "851d9759-823c-4b62-a951-c77f06430c99", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124432025-04-30T06:30:20.623ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
124442025-04-30T06:30:20.624ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fb80e90-59e9-4b6a-b392-3f38029bbacd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124452025-04-30T06:30:20.625ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
124462025-04-30T06:30:20.627ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69a46de4-e82a-459a-b158-2ffa2f3d90dc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124472025-04-30T06:30:20.628ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
124482025-04-30T06:30:20.630ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd6e5464-83cc-437a-9323-8968dc81043e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124492025-04-30T06:30:20.630ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
124502025-04-30T06:30:20.643ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63bae10f-5abf-41ea-b5ea-281b905165b4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124512025-04-30T06:30:20.643ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
124522025-04-30T06:30:20.651ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05dbeb6a-c724-4fed-bb7b-811b3b323afa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124532025-04-30T06:30:20.651ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
124542025-04-30T06:30:20.658ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
124552025-04-30T06:30:20.665ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4668774f-3e25-4761-a9db-6e2ca15283e6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124562025-04-30T06:30:20.666ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
124572025-04-30T06:30:20.669ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7f7af53-8f7c-4e75-9339-6c6c10cd4335", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124582025-04-30T06:30:20.669ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
124592025-04-30T06:30:20.671ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dcc3a69-36e9-44d1-b13a-112152b551ed", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124602025-04-30T06:30:20.671ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
124612025-04-30T06:30:20.672ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5f25c0c-ade1-4e97-95e8-515f26f4bef8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124622025-04-30T06:30:20.672ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
124632025-04-30T06:30:20.674ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d94ab901-192b-4127-bddd-696f971cffa6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124642025-04-30T06:30:20.674ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
124652025-04-30T06:30:20.675ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3fa6fe9-8dfb-44fc-86ca-9a4db4779377", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:19 GMT"} })
124662025-04-30T06:30:20.675ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
124672025-04-30T06:30:20.675ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124682025-04-30T06:30:20.676ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124692025-04-30T06:30:20.676ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124702025-04-30T06:30:20.676ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124712025-04-30T06:30:20.688ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124722025-04-30T06:30:20.715ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
124732025-04-30T06:30:20.715ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
124742025-04-30T06:30:20.715ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124752025-04-30T06:30:20.715ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124762025-04-30T06:30:20.715ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124772025-04-30T06:30:20.716ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124782025-04-30T06:30:20.717ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
124792025-04-30T06:30:20.717ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124802025-04-30T06:30:20.717ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124812025-04-30T06:30:20.717ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
124822025-04-30T06:30:20.717ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124832025-04-30T06:30:20.720ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124842025-04-30T06:30:20.734ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
124852025-04-30T06:30:20.734ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124862025-04-30T06:30:20.734ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124872025-04-30T06:30:20.735ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124882025-04-30T06:30:20.738ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124892025-04-30T06:30:20.742ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
124902025-04-30T06:30:20.742ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124912025-04-30T06:30:20.743ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
124922025-04-30T06:30:20.743ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124932025-04-30T06:30:20.746ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124942025-04-30T06:30:20.746ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
124952025-04-30T06:30:20.746ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124962025-04-30T06:30:20.810ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
124972025-04-30T06:30:20.810ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
124982025-04-30T06:30:20.811ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:42260 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
124992025-04-30T06:30:20.811ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
125002025-04-30T06:30:20.811ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
125012025-04-30T06:30:20.811ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
125022025-04-30T06:30:20.812ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
125032025-04-30T06:30:20.813ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:42260 req_id = 35d04806-93ae-4da1-a339-e9d1aa0d4e29 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
125042025-04-30T06:30:20.813ZINFOtest_image_promotion (dropshot): request completed latency_us = 499 local_addr = [::1]:60070 method = GET remote_addr = [::1]:42260 req_id = 35d04806-93ae-4da1-a339-e9d1aa0d4e29 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
125052025-04-30T06:30:20.813ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "35d04806-93ae-4da1-a339-e9d1aa0d4e29", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
125062025-04-30T06:30:20.816ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
125072025-04-30T06:30:20.816ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
125082025-04-30T06:30:20.816ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
125092025-04-30T06:30:20.818ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
125102025-04-30T06:30:20.818ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125112025-04-30T06:30:20.818ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125122025-04-30T06:30:20.818ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125132025-04-30T06:30:20.819ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125142025-04-30T06:30:20.825ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125152025-04-30T06:30:20.832ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
125162025-04-30T06:30:20.832ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125172025-04-30T06:30:20.832ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125182025-04-30T06:30:20.833ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125192025-04-30T06:30:20.835ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
125202025-04-30T06:30:20.835ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
125212025-04-30T06:30:20.839ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
125222025-04-30T06:30:20.839ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
125232025-04-30T06:30:20.862ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
125242025-04-30T06:30:20.864ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
125252025-04-30T06:30:20.864ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
125262025-04-30T06:30:20.864ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 486.823112ms iteration = 45 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
125272025-04-30T06:30:20.864ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 46 reason = Signaled
125282025-04-30T06:30:20.864ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
125292025-04-30T06:30:20.864ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125302025-04-30T06:30:20.864ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125312025-04-30T06:30:20.864ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125322025-04-30T06:30:20.865ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125332025-04-30T06:30:20.868ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125342025-04-30T06:30:20.880ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
125352025-04-30T06:30:20.880ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125362025-04-30T06:30:20.880ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125372025-04-30T06:30:20.882ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125382025-04-30T06:30:20.885ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
125392025-04-30T06:30:20.885ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
125402025-04-30T06:30:20.886ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 60 9c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 `............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59905 req_id = 58678f60-f0f0-4f2b-82cf-8ca3181874b9
125412025-04-30T06:30:20.886ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24732,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59905 req_id = 58678f60-f0f0-4f2b-82cf-8ca3181874b9
125422025-04-30T06:30:20.886ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
125432025-04-30T06:30:20.887ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125442025-04-30T06:30:20.887ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59905 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 58678f60-f0f0-4f2b-82cf-8ca3181874b9
125452025-04-30T06:30:20.887ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3485, tv_nsec: 686459186 } })
125462025-04-30T06:30:20.887ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
125472025-04-30T06:30:20.887ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5a 08 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e Z...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56255 req_id = d81d4a7b-1eaa-4d71-8c9b-ce3dcfc4b5f9
125482025-04-30T06:30:20.887ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23048,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56255 req_id = d81d4a7b-1eaa-4d71-8c9b-ce3dcfc4b5f9
125492025-04-30T06:30:20.888ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125502025-04-30T06:30:20.888ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56255 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d81d4a7b-1eaa-4d71-8c9b-ce3dcfc4b5f9
125512025-04-30T06:30:20.954ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
125522025-04-30T06:30:20.954ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125532025-04-30T06:30:20.955ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125542025-04-30T06:30:20.955ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125552025-04-30T06:30:20.955ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
125562025-04-30T06:30:20.955ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125572025-04-30T06:30:20.956ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
125582025-04-30T06:30:20.957ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 4e be 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c N............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59429 req_id = 4210bcac-4d2a-4a90-aae3-b5a62ebe9438
125592025-04-30T06:30:20.957ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20158,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59429 req_id = 4210bcac-4d2a-4a90-aae3-b5a62ebe9438
125602025-04-30T06:30:20.957ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
125612025-04-30T06:30:20.957ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20158, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59429 req_id = 4210bcac-4d2a-4a90-aae3-b5a62ebe9438
125622025-04-30T06:30:20.957ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
125632025-04-30T06:30:20.957ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
125642025-04-30T06:30:20.957ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
125652025-04-30T06:30:20.963ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125662025-04-30T06:30:20.975ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125672025-04-30T06:30:20.975ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125682025-04-30T06:30:20.975ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
125692025-04-30T06:30:20.976ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125702025-04-30T06:30:20.983ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125712025-04-30T06:30:20.984ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
125722025-04-30T06:30:20.985ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125732025-04-30T06:30:20.997ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "357b68cb-1ea8-438d-bddb-82bce588f190", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
125742025-04-30T06:30:20.998ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
125752025-04-30T06:30:21.010ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93405fb5-c8ec-4cbe-b008-c4796206c5fa", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
125762025-04-30T06:30:21.010ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
125772025-04-30T06:30:21.018ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e4e8aec-209f-4486-96bb-9d952b44c442", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
125782025-04-30T06:30:21.019ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
125792025-04-30T06:30:21.023ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9cbdb39-3758-44ee-a8e8-0b5c345c6169", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
125802025-04-30T06:30:21.024ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
125812025-04-30T06:30:21.026ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e36ac696-fbad-4805-b5ab-2d1997f10b3e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
125822025-04-30T06:30:21.027ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
125832025-04-30T06:30:21.029ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98e17922-0346-459e-a913-689513f73453", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
125842025-04-30T06:30:21.029ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
125852025-04-30T06:30:21.031ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bd3cb20-5beb-4441-897c-f8d81b54b83d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
125862025-04-30T06:30:21.031ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
125872025-04-30T06:30:21.039ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
125882025-04-30T06:30:21.041ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
125892025-04-30T06:30:21.042ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 60 95 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 `............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:51723 req_id = e610c9c6-02dd-4a0a-a0c8-996a6241b9ba
125902025-04-30T06:30:21.042ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24725,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51723 req_id = e610c9c6-02dd-4a0a-a0c8-996a6241b9ba
125912025-04-30T06:30:21.042ZDEBGtest_image_promotion (store): query key key = _dendrite._tcp
125922025-04-30T06:30:21.042ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125932025-04-30T06:30:21.042ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51723 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e610c9c6-02dd-4a0a-a0c8-996a6241b9ba
125942025-04-30T06:30:21.043ZDEBGtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3485, tv_nsec: 842183497 } })
125952025-04-30T06:30:21.043ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
125962025-04-30T06:30:21.043ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1f 22 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ."..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52848 req_id = 96a633a8-0b19-4d3c-af71-48ab0676ab52
125972025-04-30T06:30:21.043ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7970,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52848 req_id = 96a633a8-0b19-4d3c-af71-48ab0676ab52
125982025-04-30T06:30:21.044ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125992025-04-30T06:30:21.044ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52848 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 96a633a8-0b19-4d3c-af71-48ab0676ab52
126002025-04-30T06:30:21.044ZINFOtest_image_promotion (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
126012025-04-30T06:30:21.095ZINFOtest_image_promotion (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
126022025-04-30T06:30:21.095ZDEBGtest_image_promotion (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
126032025-04-30T06:30:21.095ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126042025-04-30T06:30:21.095ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126052025-04-30T06:30:21.095ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126062025-04-30T06:30:21.095ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
126072025-04-30T06:30:21.096ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126082025-04-30T06:30:21.099ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4235da79-8e1a-416a-b5a8-a2beadabfca0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126092025-04-30T06:30:21.099ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
126102025-04-30T06:30:21.099ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126112025-04-30T06:30:21.106ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126122025-04-30T06:30:21.106ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126132025-04-30T06:30:21.106ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
126142025-04-30T06:30:21.108ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126152025-04-30T06:30:21.109ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06698090-73d5-4ffd-96c9-9eabddc0f4c4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126162025-04-30T06:30:21.110ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
126172025-04-30T06:30:21.110ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126182025-04-30T06:30:21.110ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
126192025-04-30T06:30:21.111ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126202025-04-30T06:30:21.113ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f1e3260-f222-4e56-9245-5efcb7d11f94", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126212025-04-30T06:30:21.113ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
126222025-04-30T06:30:21.114ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b2d3c82-e922-48e3-be21-73508f97987f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126232025-04-30T06:30:21.115ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
126242025-04-30T06:30:21.126ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b896de8-72a2-4c34-8842-f96d9882e4bb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126252025-04-30T06:30:21.127ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
126262025-04-30T06:30:21.134ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c8cdaff-8470-45e3-add7-51d6fafc2dc2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126272025-04-30T06:30:21.135ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
126282025-04-30T06:30:21.143ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3eb1120-fa3b-429c-b878-0426fc73f668", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126292025-04-30T06:30:21.143ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
126302025-04-30T06:30:21.150ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "587366cf-382a-4160-b9c9-b994fe55ce1c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126312025-04-30T06:30:21.152ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
126322025-04-30T06:30:21.154ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
126332025-04-30T06:30:21.159ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
126342025-04-30T06:30:21.160ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: a0 f5 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:59469 req_id = 2a3eb97e-0d23-4c74-aca1-29be1a215087
126352025-04-30T06:30:21.161ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41205,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59469 req_id = 2a3eb97e-0d23-4c74-aca1-29be1a215087
126362025-04-30T06:30:21.161ZDEBGtest_image_promotion (store): query key key = _dendrite._tcp
126372025-04-30T06:30:21.162ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
126382025-04-30T06:30:21.162ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59469 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2a3eb97e-0d23-4c74-aca1-29be1a215087
126392025-04-30T06:30:21.164ZDEBGtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3485, tv_nsec: 962116966 } })
126402025-04-30T06:30:21.164ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
126412025-04-30T06:30:21.169ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ec 5d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .]..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52234 req_id = 9c4eb6cf-e846-4b90-b95b-2d417f5cf8da
126422025-04-30T06:30:21.170ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60509,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52234 req_id = 9c4eb6cf-e846-4b90-b95b-2d417f5cf8da
126432025-04-30T06:30:21.170ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
126442025-04-30T06:30:21.171ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52234 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9c4eb6cf-e846-4b90-b95b-2d417f5cf8da
126452025-04-30T06:30:21.171ZINFOtest_image_promotion (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
126462025-04-30T06:30:21.272ZINFOtest_image_promotion (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
126472025-04-30T06:30:21.273ZDEBGtest_image_promotion (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
126482025-04-30T06:30:21.273ZDEBGtest_image_promotion (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
126492025-04-30T06:30:21.273ZWARNtest_image_promotion (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
126502025-04-30T06:30:21.274ZWARNtest_image_promotion (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
126512025-04-30T06:30:21.275ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126522025-04-30T06:30:21.275ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126532025-04-30T06:30:21.275ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126542025-04-30T06:30:21.276ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
126552025-04-30T06:30:21.276ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126562025-04-30T06:30:21.299ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
126572025-04-30T06:30:21.301ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 3b 62 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ;b..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:59830 req_id = 227f76c5-ea5c-4006-b0cb-ed88dd55b902
126582025-04-30T06:30:21.301ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15202,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59830 req_id = 227f76c5-ea5c-4006-b0cb-ed88dd55b902
126592025-04-30T06:30:21.301ZDEBGtest_image_promotion (store): query key key = _dendrite._tcp
126602025-04-30T06:30:21.302ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
126612025-04-30T06:30:21.302ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59830 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 227f76c5-ea5c-4006-b0cb-ed88dd55b902
126622025-04-30T06:30:21.302ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126632025-04-30T06:30:21.313ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126642025-04-30T06:30:21.313ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126652025-04-30T06:30:21.313ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
126662025-04-30T06:30:21.313ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126672025-04-30T06:30:21.314ZDEBGtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3486, tv_nsec: 113121874 } })
126682025-04-30T06:30:21.314ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
126692025-04-30T06:30:21.316ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8b a7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53342 req_id = e7ab946e-3d78-4232-8bd9-d3ef9240dc6f
126702025-04-30T06:30:21.316ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35751,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53342 req_id = e7ab946e-3d78-4232-8bd9-d3ef9240dc6f
126712025-04-30T06:30:21.317ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
126722025-04-30T06:30:21.317ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53342 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e7ab946e-3d78-4232-8bd9-d3ef9240dc6f
126732025-04-30T06:30:21.317ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a58a6af2-97b2-4611-b4b4-2ee555cc8a65", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126742025-04-30T06:30:21.317ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
126752025-04-30T06:30:21.318ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126762025-04-30T06:30:21.318ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
126772025-04-30T06:30:21.323ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126782025-04-30T06:30:21.327ZINFOtest_image_promotion (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
126792025-04-30T06:30:21.398ZINFOtest_image_promotion (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
126802025-04-30T06:30:21.398ZDEBGtest_image_promotion (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
126812025-04-30T06:30:21.398ZDEBGtest_image_promotion (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
126822025-04-30T06:30:21.398ZWARNtest_image_promotion (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
126832025-04-30T06:30:21.399ZWARNtest_image_promotion (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
126842025-04-30T06:30:21.413ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "acf04e1a-4647-4c9d-a432-ccf3adac071e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126852025-04-30T06:30:21.414ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
126862025-04-30T06:30:21.430ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20fba1dc-6755-40c4-980e-ce38a9fbcdde", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126872025-04-30T06:30:21.430ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
126882025-04-30T06:30:21.438ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "419fd4c3-4336-4b02-8e46-35c7127231dd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126892025-04-30T06:30:21.438ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
126902025-04-30T06:30:21.441ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6ddd4c5-47c0-4ce8-8356-049b4b5bd1aa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126912025-04-30T06:30:21.441ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
126922025-04-30T06:30:21.443ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f56b3ce-0852-4938-95e5-eeaf8c5539e6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126932025-04-30T06:30:21.443ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
126942025-04-30T06:30:21.446ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78520b54-8b20-4a51-aeb8-1e5aaed643a5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126952025-04-30T06:30:21.447ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
126962025-04-30T06:30:21.448ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f598f405-418c-49ea-ac5a-d671485f14c2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126972025-04-30T06:30:21.448ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
126982025-04-30T06:30:21.451ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7623af00-e716-4dc4-9f85-40b21b2fdc09", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
126992025-04-30T06:30:21.451ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
127002025-04-30T06:30:21.454ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
127012025-04-30T06:30:21.469ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0467485-209c-43cf-9955-7aa7cc7dfc82", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127022025-04-30T06:30:21.469ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
127032025-04-30T06:30:21.471ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c9a0fdd-a162-4c1b-8220-422aede931fc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127042025-04-30T06:30:21.471ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
127052025-04-30T06:30:21.482ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37ad7ac1-2e55-4528-8833-ed4909ef2f10", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127062025-04-30T06:30:21.483ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
127072025-04-30T06:30:21.484ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e1fc9fe-51c6-41b6-b9e2-c4d8906464ca", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127082025-04-30T06:30:21.484ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
127092025-04-30T06:30:21.487ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6109264-2aa2-41cb-8cd9-f71357408ce2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127102025-04-30T06:30:21.487ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
127112025-04-30T06:30:21.489ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66ba06cc-1400-47c6-94b2-f83d4a46bf70", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127122025-04-30T06:30:21.490ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
127132025-04-30T06:30:21.491ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb2e2031-6710-4580-9de7-ebac02d5eb0f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127142025-04-30T06:30:21.492ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
127152025-04-30T06:30:21.493ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61d72239-fd6b-4334-bd8a-6aece1b669e6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127162025-04-30T06:30:21.494ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
127172025-04-30T06:30:21.495ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16cc2016-b6e4-474d-94ff-8dff1d62faac", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127182025-04-30T06:30:21.496ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
127192025-04-30T06:30:21.498ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88e0d86b-69a2-49c0-852b-814f4b788fef", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127202025-04-30T06:30:21.498ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
127212025-04-30T06:30:21.502ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69d3ea9e-942d-48f9-88ee-c9bdccd53488", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127222025-04-30T06:30:21.502ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
127232025-04-30T06:30:21.505ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127242025-04-30T06:30:21.505ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127252025-04-30T06:30:21.505ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127262025-04-30T06:30:21.505ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
127272025-04-30T06:30:21.507ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127282025-04-30T06:30:21.513ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127292025-04-30T06:30:21.521ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127302025-04-30T06:30:21.523ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127312025-04-30T06:30:21.524ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
127322025-04-30T06:30:21.525ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127332025-04-30T06:30:21.525ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab6088f1-2e21-4056-aea9-79d3d0fd29fe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127342025-04-30T06:30:21.526ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
127352025-04-30T06:30:21.530ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127362025-04-30T06:30:21.530ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
127372025-04-30T06:30:21.531ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127382025-04-30T06:30:21.535ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3238665-35ed-4c7d-9212-a06b47415500", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127392025-04-30T06:30:21.535ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
127402025-04-30T06:30:21.546ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127412025-04-30T06:30:21.546ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127422025-04-30T06:30:21.548ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63b7de77-30c2-48d5-a2ed-255dcbf8196e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127432025-04-30T06:30:21.548ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
127442025-04-30T06:30:21.550ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127452025-04-30T06:30:21.552ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127462025-04-30T06:30:21.552ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127472025-04-30T06:30:21.552ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127482025-04-30T06:30:21.555ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66e75a62-dee7-46d4-a36d-82ebe7325a7a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127492025-04-30T06:30:21.556ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
127502025-04-30T06:30:21.563ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "439cee6a-a65e-4434-8e13-0c16bd6195c4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127512025-04-30T06:30:21.563ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
127522025-04-30T06:30:21.570ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5c3571c-61a5-4d11-89f3-005b40a04188", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127532025-04-30T06:30:21.570ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
127542025-04-30T06:30:21.571ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38d6de01-175e-4c64-bff0-58e050273605", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127552025-04-30T06:30:21.571ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
127562025-04-30T06:30:21.574ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9e3e0a3-ab29-4ded-a804-ad3ba8a95851", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127572025-04-30T06:30:21.574ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
127582025-04-30T06:30:21.575ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "527b8ea9-77fa-41e0-af20-d87c3a33ab73", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127592025-04-30T06:30:21.575ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
127602025-04-30T06:30:21.575ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
127612025-04-30T06:30:21.577ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73697c9a-c6e8-4d57-b85e-d0873956c563", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127622025-04-30T06:30:21.577ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
127632025-04-30T06:30:21.577ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127642025-04-30T06:30:21.577ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127652025-04-30T06:30:21.577ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127662025-04-30T06:30:21.577ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127672025-04-30T06:30:21.582ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127682025-04-30T06:30:21.586ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
127692025-04-30T06:30:21.587ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
127702025-04-30T06:30:21.587ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127712025-04-30T06:30:21.587ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127722025-04-30T06:30:21.587ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127732025-04-30T06:30:21.588ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127742025-04-30T06:30:21.591ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127752025-04-30T06:30:21.596ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
127762025-04-30T06:30:21.596ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127772025-04-30T06:30:21.596ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127782025-04-30T06:30:21.597ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127792025-04-30T06:30:21.656ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
127802025-04-30T06:30:21.656ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
127812025-04-30T06:30:21.657ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127822025-04-30T06:30:21.657ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127832025-04-30T06:30:21.657ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127842025-04-30T06:30:21.657ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
127852025-04-30T06:30:21.658ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127862025-04-30T06:30:21.658ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:35504 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
127872025-04-30T06:30:21.659ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:35504 req_id = 9ede5786-5d37-42d7-8837-816da16500a7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
127882025-04-30T06:30:21.660ZINFOtest_image_promotion (dropshot): request completed latency_us = 653 local_addr = [::1]:60070 method = GET remote_addr = [::1]:35504 req_id = 9ede5786-5d37-42d7-8837-816da16500a7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
127892025-04-30T06:30:21.660ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ede5786-5d37-42d7-8837-816da16500a7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:20 GMT"} })
127902025-04-30T06:30:21.663ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
127912025-04-30T06:30:21.663ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
127922025-04-30T06:30:21.663ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
127932025-04-30T06:30:21.666ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
127942025-04-30T06:30:21.666ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127952025-04-30T06:30:21.666ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127962025-04-30T06:30:21.666ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127972025-04-30T06:30:21.667ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127982025-04-30T06:30:21.668ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127992025-04-30T06:30:21.676ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128002025-04-30T06:30:21.676ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128012025-04-30T06:30:21.676ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
128022025-04-30T06:30:21.677ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128032025-04-30T06:30:21.682ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128042025-04-30T06:30:21.693ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
128052025-04-30T06:30:21.693ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128062025-04-30T06:30:21.694ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128072025-04-30T06:30:21.694ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128082025-04-30T06:30:21.695ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128092025-04-30T06:30:21.695ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
128102025-04-30T06:30:21.695ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128112025-04-30T06:30:21.735ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
128122025-04-30T06:30:21.735ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
128132025-04-30T06:30:21.735ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 871.087786ms iteration = 46 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
128142025-04-30T06:30:21.735ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 47 reason = Signaled
128152025-04-30T06:30:21.735ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
128162025-04-30T06:30:21.735ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128172025-04-30T06:30:21.735ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128182025-04-30T06:30:21.736ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128192025-04-30T06:30:21.736ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128202025-04-30T06:30:21.747ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
128212025-04-30T06:30:21.747ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128222025-04-30T06:30:21.752ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
128232025-04-30T06:30:21.752ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128242025-04-30T06:30:21.752ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128252025-04-30T06:30:21.752ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128262025-04-30T06:30:21.753ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
128272025-04-30T06:30:21.753ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
128282025-04-30T06:30:21.753ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
128292025-04-30T06:30:21.754ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
128302025-04-30T06:30:21.759ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
128312025-04-30T06:30:21.759ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
128322025-04-30T06:30:21.761ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
128332025-04-30T06:30:21.761ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
128342025-04-30T06:30:21.763ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 69 67 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ig..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62052 req_id = b4dd2c2f-39d6-4515-9236-5d31ddcd577e
128352025-04-30T06:30:21.763ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26983,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62052 req_id = b4dd2c2f-39d6-4515-9236-5d31ddcd577e
128362025-04-30T06:30:21.763ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
128372025-04-30T06:30:21.763ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
128382025-04-30T06:30:21.763ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62052 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b4dd2c2f-39d6-4515-9236-5d31ddcd577e
128392025-04-30T06:30:21.764ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3486, tv_nsec: 563386647 } })
128402025-04-30T06:30:21.764ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
128412025-04-30T06:30:21.765ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
128422025-04-30T06:30:21.765ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
128432025-04-30T06:30:21.765ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 03 57 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .W..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52713 req_id = 3f0c828b-d33e-49e1-a929-0aa29c418438
128442025-04-30T06:30:21.765ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 855,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52713 req_id = 3f0c828b-d33e-49e1-a929-0aa29c418438
128452025-04-30T06:30:21.765ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
128462025-04-30T06:30:21.765ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52713 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3f0c828b-d33e-49e1-a929-0aa29c418438
128472025-04-30T06:30:21.809ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
128482025-04-30T06:30:21.809ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128492025-04-30T06:30:21.810ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128502025-04-30T06:30:21.810ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128512025-04-30T06:30:21.810ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
128522025-04-30T06:30:21.812ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128532025-04-30T06:30:21.814ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2c 74 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ,t..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63973 req_id = d1c97347-e0c5-4859-91f4-383fabd180b0
128542025-04-30T06:30:21.814ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11380,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63973 req_id = d1c97347-e0c5-4859-91f4-383fabd180b0
128552025-04-30T06:30:21.815ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
128562025-04-30T06:30:21.815ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11380, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63973 req_id = d1c97347-e0c5-4859-91f4-383fabd180b0
128572025-04-30T06:30:21.815ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
128582025-04-30T06:30:21.815ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
128592025-04-30T06:30:21.815ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
128602025-04-30T06:30:21.817ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128612025-04-30T06:30:21.834ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128622025-04-30T06:30:21.834ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128632025-04-30T06:30:21.835ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
128642025-04-30T06:30:21.836ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128652025-04-30T06:30:21.842ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7efa0f8-76f8-467f-8f93-8ac729533877", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128662025-04-30T06:30:21.842ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
128672025-04-30T06:30:21.842ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128682025-04-30T06:30:21.843ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
128692025-04-30T06:30:21.844ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128702025-04-30T06:30:21.850ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bf2defe-a764-4d3a-9f14-b1e154a237f5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128712025-04-30T06:30:21.850ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
128722025-04-30T06:30:21.855ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
128732025-04-30T06:30:21.859ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f43473a-26dd-4f6f-ac0f-eca33df12cae", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128742025-04-30T06:30:21.859ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
128752025-04-30T06:30:21.864ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fafb9476-2b85-43f9-9f99-0729a8c59ed5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128762025-04-30T06:30:21.864ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
128772025-04-30T06:30:21.871ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43bdc99c-aff3-4310-88b8-348b66f35982", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128782025-04-30T06:30:21.871ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
128792025-04-30T06:30:21.876ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
128802025-04-30T06:30:21.879ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28f239fb-ac31-4b36-b8a7-40a555f6523d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128812025-04-30T06:30:21.880ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
128822025-04-30T06:30:21.883ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e0d8c7c-bc5a-412f-a292-6c33f6783a1f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128832025-04-30T06:30:21.883ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
128842025-04-30T06:30:21.887ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03268093-03cb-48e6-9404-24a20ff72e2d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128852025-04-30T06:30:21.887ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
128862025-04-30T06:30:21.891ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6374109-6a56-491b-a1f7-331a73d849bd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128872025-04-30T06:30:21.891ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
128882025-04-30T06:30:21.893ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7331a010-42e4-419c-b79c-7b99a412c5d1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128892025-04-30T06:30:21.893ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
128902025-04-30T06:30:21.895ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19f8d505-dd77-48c7-933c-47a0c3feff3d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128912025-04-30T06:30:21.895ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
128922025-04-30T06:30:21.896ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7638d70-8e34-4f6a-a2b3-6b9f2bf558b8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128932025-04-30T06:30:21.896ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
128942025-04-30T06:30:21.899ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c570137d-1605-4408-9f9f-3ab5220ab612", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128952025-04-30T06:30:21.900ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
128962025-04-30T06:30:21.906ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b786fe76-5780-4a23-bcef-d963df42bb8d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128972025-04-30T06:30:21.907ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
128982025-04-30T06:30:21.910ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75bf2e3d-e231-4f55-a5a4-20bda748b9cd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
128992025-04-30T06:30:21.910ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
129002025-04-30T06:30:21.913ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "142610b6-f28e-4ffd-87b6-31a9d7f9c3e2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129012025-04-30T06:30:21.913ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
129022025-04-30T06:30:21.916ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3df2be0-bf73-49b4-9d93-e66a2d232f45", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129032025-04-30T06:30:21.916ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
129042025-04-30T06:30:21.918ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "691c7181-a256-47fc-9170-cfdb550b6bce", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129052025-04-30T06:30:21.918ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
129062025-04-30T06:30:21.920ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "155e71b8-c4d6-4e13-be86-3f85d606bbfd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129072025-04-30T06:30:21.920ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
129082025-04-30T06:30:21.922ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "854ee999-b67d-4bec-9839-56b100761f2c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129092025-04-30T06:30:21.922ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
129102025-04-30T06:30:21.924ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6faedf84-b0dd-4ce1-bbd2-0930f6e292ab", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129112025-04-30T06:30:21.924ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
129122025-04-30T06:30:21.925ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0adb62c-05b6-46f6-807b-d23294f785dd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129132025-04-30T06:30:21.925ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
129142025-04-30T06:30:21.926ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18feae23-32fc-4ba1-87c9-b28dc8d8d6f3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129152025-04-30T06:30:21.926ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
129162025-04-30T06:30:21.926ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
129172025-04-30T06:30:21.927ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129182025-04-30T06:30:21.927ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129192025-04-30T06:30:21.927ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
129202025-04-30T06:30:21.928ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129212025-04-30T06:30:21.931ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b5d9af2-0301-4d7a-9bc2-e22141d26ef7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129222025-04-30T06:30:21.931ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
129232025-04-30T06:30:21.931ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129242025-04-30T06:30:21.938ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129252025-04-30T06:30:21.938ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129262025-04-30T06:30:21.938ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
129272025-04-30T06:30:21.938ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129282025-04-30T06:30:21.941ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccdca952-08c4-486b-9221-80de81721b64", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129292025-04-30T06:30:21.941ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
129302025-04-30T06:30:21.941ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129312025-04-30T06:30:21.941ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
129322025-04-30T06:30:21.942ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129332025-04-30T06:30:21.944ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1598b66e-61c4-4f42-9cd3-55d79d304a5a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129342025-04-30T06:30:21.944ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
129352025-04-30T06:30:21.947ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b6ec598-a396-4604-b553-1299a4169469", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129362025-04-30T06:30:21.947ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
129372025-04-30T06:30:21.962ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3288acb0-3821-49c1-982d-4fc6e495e609", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129382025-04-30T06:30:21.962ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
129392025-04-30T06:30:22.003ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
129402025-04-30T06:30:22.014ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "214cfe2c-8018-468f-9fb3-ad79b0d1a6aa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129412025-04-30T06:30:22.014ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
129422025-04-30T06:30:22.018ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43301369-a2c5-4272-b4d1-3ce7f1d7e034", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129432025-04-30T06:30:22.019ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
129442025-04-30T06:30:22.020ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6df5dce5-2f6c-4f02-b47a-c0ba8eb596aa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129452025-04-30T06:30:22.020ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
129462025-04-30T06:30:22.021ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d48f617-085e-4894-ad55-72b36cee2651", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129472025-04-30T06:30:22.022ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
129482025-04-30T06:30:22.023ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fdf47cd-65d4-4d64-a80a-b5a5e04ab3cb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129492025-04-30T06:30:22.023ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
129502025-04-30T06:30:22.024ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2db28ddf-988c-490a-8d85-3d584f434dfd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129512025-04-30T06:30:22.024ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
129522025-04-30T06:30:22.026ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac7554a5-3486-44de-8644-fee56b517829", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129532025-04-30T06:30:22.026ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
129542025-04-30T06:30:22.028ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b646730d-fe46-4cea-887a-9ee2e1dadc41", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129552025-04-30T06:30:22.028ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
129562025-04-30T06:30:22.030ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25a40a75-7bdb-41c2-8ef6-9a186e39e885", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129572025-04-30T06:30:22.030ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
129582025-04-30T06:30:22.033ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a90db82-2b81-402b-99c8-49edb26526c4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129592025-04-30T06:30:22.033ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
129602025-04-30T06:30:22.035ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e984834-9199-41d4-bb72-35985818c672", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129612025-04-30T06:30:22.035ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
129622025-04-30T06:30:22.039ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b4d6142-d0e8-4b3e-b5b8-6f659a843207", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129632025-04-30T06:30:22.039ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
129642025-04-30T06:30:22.041ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a058cb4b-9047-42ef-87be-6ce60db53e3a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129652025-04-30T06:30:22.041ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
129662025-04-30T06:30:22.042ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5ed9887-0595-45a4-b7ab-e0c137f4e120", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129672025-04-30T06:30:22.042ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
129682025-04-30T06:30:22.043ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "213327ee-b31b-4685-8e85-217edf8339fb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129692025-04-30T06:30:22.043ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
129702025-04-30T06:30:22.044ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe6a92d8-08dd-4fe2-95c4-c535f70aceec", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129712025-04-30T06:30:22.044ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
129722025-04-30T06:30:22.045ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94a7d9f8-1cfa-433a-bde4-374f42805ba2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
129732025-04-30T06:30:22.045ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
129742025-04-30T06:30:22.045ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129752025-04-30T06:30:22.045ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129762025-04-30T06:30:22.046ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129772025-04-30T06:30:22.046ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129782025-04-30T06:30:22.049ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129792025-04-30T06:30:22.052ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
129802025-04-30T06:30:22.052ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
129812025-04-30T06:30:22.052ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129822025-04-30T06:30:22.052ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129832025-04-30T06:30:22.052ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129842025-04-30T06:30:22.053ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129852025-04-30T06:30:22.054ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
129862025-04-30T06:30:22.054ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129872025-04-30T06:30:22.054ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129882025-04-30T06:30:22.054ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
129892025-04-30T06:30:22.054ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129902025-04-30T06:30:22.056ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129912025-04-30T06:30:22.062ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
129922025-04-30T06:30:22.062ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129932025-04-30T06:30:22.062ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129942025-04-30T06:30:22.063ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129952025-04-30T06:30:22.064ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129962025-04-30T06:30:22.070ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129972025-04-30T06:30:22.070ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129982025-04-30T06:30:22.070ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
129992025-04-30T06:30:22.070ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130002025-04-30T06:30:22.073ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130012025-04-30T06:30:22.073ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
130022025-04-30T06:30:22.074ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130032025-04-30T06:30:22.136ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
130042025-04-30T06:30:22.136ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
130052025-04-30T06:30:22.137ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:40206 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130062025-04-30T06:30:22.137ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:40206 req_id = 5800f234-fb02-4ee6-a1f5-fef0e5aaa9d9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
130072025-04-30T06:30:22.137ZINFOtest_image_promotion (dropshot): request completed latency_us = 473 local_addr = [::1]:60070 method = GET remote_addr = [::1]:40206 req_id = 5800f234-fb02-4ee6-a1f5-fef0e5aaa9d9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
130082025-04-30T06:30:22.139ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5800f234-fb02-4ee6-a1f5-fef0e5aaa9d9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130092025-04-30T06:30:22.141ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
130102025-04-30T06:30:22.141ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
130112025-04-30T06:30:22.141ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
130122025-04-30T06:30:22.144ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
130132025-04-30T06:30:22.144ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130142025-04-30T06:30:22.144ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130152025-04-30T06:30:22.144ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130162025-04-30T06:30:22.145ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130172025-04-30T06:30:22.156ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130182025-04-30T06:30:22.161ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
130192025-04-30T06:30:22.162ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130202025-04-30T06:30:22.162ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130212025-04-30T06:30:22.162ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130222025-04-30T06:30:22.199ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
130232025-04-30T06:30:22.204ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
130242025-04-30T06:30:22.206ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
130252025-04-30T06:30:22.207ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 471.269763ms iteration = 47 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
130262025-04-30T06:30:22.207ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 48 reason = Signaled
130272025-04-30T06:30:22.207ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
130282025-04-30T06:30:22.207ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130292025-04-30T06:30:22.207ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130302025-04-30T06:30:22.207ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130312025-04-30T06:30:22.210ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130322025-04-30T06:30:22.213ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130332025-04-30T06:30:22.225ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
130342025-04-30T06:30:22.226ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130352025-04-30T06:30:22.226ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130362025-04-30T06:30:22.227ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130372025-04-30T06:30:22.228ZDEBGtest_image_promotion (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
130382025-04-30T06:30:22.228ZWARNtest_image_promotion (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
130392025-04-30T06:30:22.230ZWARNtest_image_promotion (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
130402025-04-30T06:30:22.233ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
130412025-04-30T06:30:22.233ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
130422025-04-30T06:30:22.234ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a5 70 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .p..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63614 req_id = dd5263ab-d436-407f-9f2c-53525b4cb646
130432025-04-30T06:30:22.234ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42352,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63614 req_id = dd5263ab-d436-407f-9f2c-53525b4cb646
130442025-04-30T06:30:22.235ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
130452025-04-30T06:30:22.235ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130462025-04-30T06:30:22.235ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63614 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = dd5263ab-d436-407f-9f2c-53525b4cb646
130472025-04-30T06:30:22.235ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3487, tv_nsec: 34521201 } })
130482025-04-30T06:30:22.235ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
130492025-04-30T06:30:22.235ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 35 f1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 5...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58322 req_id = d5cf074f-a2ef-4ce8-922d-3262b145566e
130502025-04-30T06:30:22.236ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13809,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58322 req_id = d5cf074f-a2ef-4ce8-922d-3262b145566e
130512025-04-30T06:30:22.236ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130522025-04-30T06:30:22.236ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58322 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d5cf074f-a2ef-4ce8-922d-3262b145566e
130532025-04-30T06:30:22.301ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
130542025-04-30T06:30:22.302ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130552025-04-30T06:30:22.302ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130562025-04-30T06:30:22.302ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130572025-04-30T06:30:22.302ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
130582025-04-30T06:30:22.303ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130592025-04-30T06:30:22.305ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 96 86 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53487 req_id = 909e79df-0e23-457d-b713-9408f9489857
130602025-04-30T06:30:22.307ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38534,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53487 req_id = 909e79df-0e23-457d-b713-9408f9489857
130612025-04-30T06:30:22.307ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
130622025-04-30T06:30:22.307ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38534, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53487 req_id = 909e79df-0e23-457d-b713-9408f9489857
130632025-04-30T06:30:22.308ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
130642025-04-30T06:30:22.308ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
130652025-04-30T06:30:22.308ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
130662025-04-30T06:30:22.309ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130672025-04-30T06:30:22.316ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130682025-04-30T06:30:22.317ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130692025-04-30T06:30:22.317ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
130702025-04-30T06:30:22.317ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130712025-04-30T06:30:22.321ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c4b9d6e-4e9c-40c2-845b-98b4c77621a3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130722025-04-30T06:30:22.322ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
130732025-04-30T06:30:22.322ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130742025-04-30T06:30:22.323ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
130752025-04-30T06:30:22.325ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130762025-04-30T06:30:22.328ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5607234b-4dba-4a3a-9b0c-a5848089928f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130772025-04-30T06:30:22.328ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
130782025-04-30T06:30:22.331ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67a6b3ab-f5cf-478d-9563-99bc9f57b9bc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130792025-04-30T06:30:22.331ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
130802025-04-30T06:30:22.334ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12652a76-e279-4652-8fda-359beed74905", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130812025-04-30T06:30:22.334ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
130822025-04-30T06:30:22.336ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1070522-2624-4a21-84e3-4c8d32872772", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130832025-04-30T06:30:22.336ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
130842025-04-30T06:30:22.340ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cfc5d96-4f22-4ffb-959e-8099f2f1c293", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130852025-04-30T06:30:22.340ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
130862025-04-30T06:30:22.344ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f69cbb3-7a4c-4c75-a0b6-89d231556ddf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130872025-04-30T06:30:22.344ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
130882025-04-30T06:30:22.346ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5382b3b0-c162-49ae-9e0e-fb4210270e28", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130892025-04-30T06:30:22.346ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
130902025-04-30T06:30:22.347ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b7433cd-0839-45f1-afee-0c5875bace24", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130912025-04-30T06:30:22.349ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
130922025-04-30T06:30:22.351ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e95be939-4f0d-489f-b90f-f5d72e0dd1d9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130932025-04-30T06:30:22.351ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
130942025-04-30T06:30:22.353ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4335a88c-a7d7-4b68-bf51-dc32481e69b4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130952025-04-30T06:30:22.354ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
130962025-04-30T06:30:22.355ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0692657-d4bb-4c44-b2b5-4b2029453021", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130972025-04-30T06:30:22.355ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
130982025-04-30T06:30:22.358ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96719ac5-4a48-41d0-a96f-70bca413b5aa", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
130992025-04-30T06:30:22.358ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
131002025-04-30T06:30:22.361ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
131012025-04-30T06:30:22.363ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fe26974-e7c8-417d-8db8-4fd4abcdc4c8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131022025-04-30T06:30:22.364ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
131032025-04-30T06:30:22.366ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e3e53fb-0626-4442-995b-7dfee6588767", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131042025-04-30T06:30:22.366ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
131052025-04-30T06:30:22.369ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "40b32e66-ea9f-4f76-919a-00bd40a4031f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131062025-04-30T06:30:22.369ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
131072025-04-30T06:30:22.372ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "552e75cf-0eda-4ba1-95ad-ccaf9c07d2a3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131082025-04-30T06:30:22.372ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
131092025-04-30T06:30:22.374ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65231eef-a53f-4d74-86bf-31e5fdd3067b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131102025-04-30T06:30:22.374ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
131112025-04-30T06:30:22.377ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36f6e267-2f04-4cbf-af68-e74c95b5c3dd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131122025-04-30T06:30:22.377ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
131132025-04-30T06:30:22.379ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b8ace98-6590-4e00-9f49-817a4cbeac4d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131142025-04-30T06:30:22.379ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
131152025-04-30T06:30:22.380ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6017e202-445f-4b66-9023-3d5971f17131", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131162025-04-30T06:30:22.380ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
131172025-04-30T06:30:22.381ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5083ea65-d511-463e-828b-3a80a7e4d798", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131182025-04-30T06:30:22.381ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
131192025-04-30T06:30:22.384ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d713c4e2-af55-411b-a96b-d6c87f734fc3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131202025-04-30T06:30:22.385ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
131212025-04-30T06:30:22.387ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a29f5022-2adc-4169-ac83-007f315b0bd5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131222025-04-30T06:30:22.387ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
131232025-04-30T06:30:22.392ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ef9d2b1-97d5-4212-95f3-d06a9e6b73a9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131242025-04-30T06:30:22.392ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
131252025-04-30T06:30:22.395ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee90e9b6-c154-416f-a65d-ac4c2cb1e704", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131262025-04-30T06:30:22.395ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
131272025-04-30T06:30:22.397ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "550871a6-a2c1-45c2-a446-dfab2aeaf6fc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131282025-04-30T06:30:22.397ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
131292025-04-30T06:30:22.400ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c84ee8ce-7a0e-4df2-b63f-ed2c4e0c60d2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131302025-04-30T06:30:22.400ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
131312025-04-30T06:30:22.404ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eccac065-e320-48cf-a63e-d3451bbf89a5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131322025-04-30T06:30:22.404ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
131332025-04-30T06:30:22.406ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "862f40fc-14a3-4b68-bb59-60ca78ef641f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131342025-04-30T06:30:22.406ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
131352025-04-30T06:30:22.407ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1394fa7e-8ac0-44c8-8c10-6ef5721f3e70", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131362025-04-30T06:30:22.408ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
131372025-04-30T06:30:22.409ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9b4633d-14fc-4f2b-b4c7-88a48818a5b6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131382025-04-30T06:30:22.409ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
131392025-04-30T06:30:22.410ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2757f361-206f-4452-8915-e106cdd455bb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131402025-04-30T06:30:22.410ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
131412025-04-30T06:30:22.412ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
131422025-04-30T06:30:22.412ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131432025-04-30T06:30:22.412ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131442025-04-30T06:30:22.412ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
131452025-04-30T06:30:22.413ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131462025-04-30T06:30:22.414ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2e2af6e-81c8-4e88-b51a-3d90e035fdf2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131472025-04-30T06:30:22.414ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
131482025-04-30T06:30:22.416ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bca85e5b-5d39-40dd-9637-536802c09791", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131492025-04-30T06:30:22.417ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
131502025-04-30T06:30:22.417ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131512025-04-30T06:30:22.424ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131522025-04-30T06:30:22.424ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131532025-04-30T06:30:22.424ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
131542025-04-30T06:30:22.425ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131552025-04-30T06:30:22.436ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131562025-04-30T06:30:22.437ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
131572025-04-30T06:30:22.437ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131582025-04-30T06:30:22.439ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ebf6f11-c427-4ad9-a85c-ae6d41a34f2c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131592025-04-30T06:30:22.439ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
131602025-04-30T06:30:22.444ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b5f04f4-dd83-462d-861b-c49d1ecf7e4b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131612025-04-30T06:30:22.444ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
131622025-04-30T06:30:22.447ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b00d3369-cad4-40ba-b260-894b25f4450f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131632025-04-30T06:30:22.447ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
131642025-04-30T06:30:22.450ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e273e5e-6931-4816-b09f-f50d07425fda", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131652025-04-30T06:30:22.450ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
131662025-04-30T06:30:22.454ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4747e980-4238-4e74-9ce9-db2eb0193986", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131672025-04-30T06:30:22.454ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
131682025-04-30T06:30:22.458ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32f9af1a-d733-4d46-8c59-3c20b84a6290", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131692025-04-30T06:30:22.458ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
131702025-04-30T06:30:22.462ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcddbf21-fdfb-4cc9-a99e-4bb7ea21134f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131712025-04-30T06:30:22.463ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
131722025-04-30T06:30:22.465ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0d86625-d30e-472f-bf85-952ceccf3a61", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131732025-04-30T06:30:22.465ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
131742025-04-30T06:30:22.468ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
131752025-04-30T06:30:22.469ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57856601-1694-4bf7-b19a-42d7127b52fb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131762025-04-30T06:30:22.470ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
131772025-04-30T06:30:22.471ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f3d7207-9620-4a49-aa56-08f39fa796b8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
131782025-04-30T06:30:22.471ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
131792025-04-30T06:30:22.471ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131802025-04-30T06:30:22.471ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131812025-04-30T06:30:22.471ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131822025-04-30T06:30:22.472ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131832025-04-30T06:30:22.475ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131842025-04-30T06:30:22.482ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
131852025-04-30T06:30:22.482ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
131862025-04-30T06:30:22.482ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131872025-04-30T06:30:22.482ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131882025-04-30T06:30:22.482ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131892025-04-30T06:30:22.482ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131902025-04-30T06:30:22.487ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131912025-04-30T06:30:22.493ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
131922025-04-30T06:30:22.493ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131932025-04-30T06:30:22.493ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131942025-04-30T06:30:22.493ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131952025-04-30T06:30:22.544ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
131962025-04-30T06:30:22.544ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
131972025-04-30T06:30:22.545ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
131982025-04-30T06:30:22.545ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131992025-04-30T06:30:22.545ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132002025-04-30T06:30:22.545ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
132012025-04-30T06:30:22.545ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132022025-04-30T06:30:22.545ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:62236 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
132032025-04-30T06:30:22.546ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
132042025-04-30T06:30:22.546ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132052025-04-30T06:30:22.546ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132062025-04-30T06:30:22.547ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132072025-04-30T06:30:22.547ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fbf6c886-662d-4e09-929d-9c1d9d295fc2 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132082025-04-30T06:30:22.548ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:62236 req_id = 01d93b85-80c1-414b-93ba-40b28a6c55cd sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
132092025-04-30T06:30:22.548ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fbf6c886-662d-4e09-929d-9c1d9d295fc2 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132102025-04-30T06:30:22.548ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fbf6c886-662d-4e09-929d-9c1d9d295fc2 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132112025-04-30T06:30:22.549ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fbf6c886-662d-4e09-929d-9c1d9d295fc2 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132122025-04-30T06:30:22.550ZINFOtest_image_promotion (dropshot): request completed latency_us = 2525 local_addr = [::1]:60070 method = GET remote_addr = [::1]:62236 req_id = 01d93b85-80c1-414b-93ba-40b28a6c55cd response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
132132025-04-30T06:30:22.551ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132142025-04-30T06:30:22.556ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
132152025-04-30T06:30:22.556ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132162025-04-30T06:30:22.556ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
132172025-04-30T06:30:22.556ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132182025-04-30T06:30:22.557ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132192025-04-30T06:30:22.557ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132202025-04-30T06:30:22.557ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132212025-04-30T06:30:22.557ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132222025-04-30T06:30:22.557ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "01d93b85-80c1-414b-93ba-40b28a6c55cd", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
132232025-04-30T06:30:22.559ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
132242025-04-30T06:30:22.559ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
132252025-04-30T06:30:22.559ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
132262025-04-30T06:30:22.561ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
132272025-04-30T06:30:22.561ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132282025-04-30T06:30:22.561ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132292025-04-30T06:30:22.561ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132302025-04-30T06:30:22.561ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132312025-04-30T06:30:22.564ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 16492 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = fbf6c886-662d-4e09-929d-9c1d9d295fc2 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132322025-04-30T06:30:22.564ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbf6c886-662d-4e09-929d-9c1d9d295fc2", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
132332025-04-30T06:30:22.564ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
132342025-04-30T06:30:22.565ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = db940a33-3738-4c03-b3f1-229a7f65a22e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
132352025-04-30T06:30:22.565ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = db940a33-3738-4c03-b3f1-229a7f65a22e resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
132362025-04-30T06:30:22.565ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = db940a33-3738-4c03-b3f1-229a7f65a22e roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
132372025-04-30T06:30:22.566ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = db940a33-3738-4c03-b3f1-229a7f65a22e resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
132382025-04-30T06:30:22.567ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132392025-04-30T06:30:22.567ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
132402025-04-30T06:30:22.568ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132412025-04-30T06:30:22.568ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132422025-04-30T06:30:22.573ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
132432025-04-30T06:30:22.573ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132442025-04-30T06:30:22.573ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132452025-04-30T06:30:22.574ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132462025-04-30T06:30:22.577ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 11855 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = db940a33-3738-4c03-b3f1-229a7f65a22e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
132472025-04-30T06:30:22.578ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "db940a33-3738-4c03-b3f1-229a7f65a22e", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
132482025-04-30T06:30:22.578ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132492025-04-30T06:30:22.579ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
132502025-04-30T06:30:22.579ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
132512025-04-30T06:30:22.629ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
132522025-04-30T06:30:22.629ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
132532025-04-30T06:30:22.629ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 422.041761ms iteration = 48 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
132542025-04-30T06:30:22.629ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 49 reason = Signaled
132552025-04-30T06:30:22.629ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
132562025-04-30T06:30:22.629ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132572025-04-30T06:30:22.629ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132582025-04-30T06:30:22.629ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132592025-04-30T06:30:22.630ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132602025-04-30T06:30:22.631ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
132612025-04-30T06:30:22.632ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132622025-04-30T06:30:22.638ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
132632025-04-30T06:30:22.638ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132642025-04-30T06:30:22.639ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132652025-04-30T06:30:22.641ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132662025-04-30T06:30:22.644ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
132672025-04-30T06:30:22.644ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
132682025-04-30T06:30:22.644ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a9 ec 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58755 req_id = 7efb06f1-76fd-44d2-a693-406c7ddc5e28
132692025-04-30T06:30:22.645ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43500,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58755 req_id = 7efb06f1-76fd-44d2-a693-406c7ddc5e28
132702025-04-30T06:30:22.645ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
132712025-04-30T06:30:22.645ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132722025-04-30T06:30:22.645ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58755 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7efb06f1-76fd-44d2-a693-406c7ddc5e28
132732025-04-30T06:30:22.645ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3487, tv_nsec: 444758983 } })
132742025-04-30T06:30:22.645ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
132752025-04-30T06:30:22.646ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ea c9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50153 req_id = a1a96ee7-ac0a-4ace-af7a-b809de24bf7d
132762025-04-30T06:30:22.646ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60105,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50153 req_id = a1a96ee7-ac0a-4ace-af7a-b809de24bf7d
132772025-04-30T06:30:22.646ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132782025-04-30T06:30:22.646ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50153 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a1a96ee7-ac0a-4ace-af7a-b809de24bf7d
132792025-04-30T06:30:22.683ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
132802025-04-30T06:30:22.683ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
132812025-04-30T06:30:22.683ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132822025-04-30T06:30:22.683ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132832025-04-30T06:30:22.683ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
132842025-04-30T06:30:22.684ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132852025-04-30T06:30:22.684ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f7 10 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59068 req_id = 8a919d92-39eb-46bf-a798-7ac697a8dfc1
132862025-04-30T06:30:22.684ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63248,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59068 req_id = 8a919d92-39eb-46bf-a798-7ac697a8dfc1
132872025-04-30T06:30:22.684ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
132882025-04-30T06:30:22.685ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63248, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59068 req_id = 8a919d92-39eb-46bf-a798-7ac697a8dfc1
132892025-04-30T06:30:22.685ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
132902025-04-30T06:30:22.685ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
132912025-04-30T06:30:22.685ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
132922025-04-30T06:30:22.687ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132932025-04-30T06:30:22.695ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
132942025-04-30T06:30:22.695ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132952025-04-30T06:30:22.695ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
132962025-04-30T06:30:22.695ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132972025-04-30T06:30:22.697ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b57b01f-be22-42f7-b47f-fd803383d9d2", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
132982025-04-30T06:30:22.697ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
132992025-04-30T06:30:22.698ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133002025-04-30T06:30:22.698ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
133012025-04-30T06:30:22.698ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133022025-04-30T06:30:22.700ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae3eb917-653c-4e2e-9894-a734b1fd78ff", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133032025-04-30T06:30:22.701ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
133042025-04-30T06:30:22.703ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c25e36b7-d736-4723-a8f4-9d1ae5855fcd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133052025-04-30T06:30:22.703ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
133062025-04-30T06:30:22.706ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39ef4e02-3588-4009-afe3-cf667afef816", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133072025-04-30T06:30:22.706ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
133082025-04-30T06:30:22.708ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4931b00f-9a5d-4c4a-b05a-b82842b86a4a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133092025-04-30T06:30:22.708ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
133102025-04-30T06:30:22.712ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a23ad0a4-01da-4e89-865a-9250dce2620f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133112025-04-30T06:30:22.712ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
133122025-04-30T06:30:22.720ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "579ef0ac-5204-42fb-af77-7631cc7e9949", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133132025-04-30T06:30:22.720ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
133142025-04-30T06:30:22.727ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ba8c0ee-8f0e-4a6d-8bc0-99f8a3a49dd1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133152025-04-30T06:30:22.727ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
133162025-04-30T06:30:22.729ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ae81264-8e67-43fd-ab55-770b84d53875", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133172025-04-30T06:30:22.729ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
133182025-04-30T06:30:22.733ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c8cfd5c-7732-4d8f-88ce-2bd4b954428d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133192025-04-30T06:30:22.733ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
133202025-04-30T06:30:22.734ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
133212025-04-30T06:30:22.735ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5da20de4-6b31-440d-8f9a-ea68838633c5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133222025-04-30T06:30:22.736ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
133232025-04-30T06:30:22.737ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50511a19-de8d-4aff-ad50-a9424e7a4dad", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133242025-04-30T06:30:22.737ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
133252025-04-30T06:30:22.739ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ca69276-127d-43d1-a1a9-87a5d85ae23f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133262025-04-30T06:30:22.739ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
133272025-04-30T06:30:22.745ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05f3c740-eeee-4886-8209-72cbf62e092b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133282025-04-30T06:30:22.745ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
133292025-04-30T06:30:22.748ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f6ef1d6-4f2e-4177-9c03-f07a3754b8eb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133302025-04-30T06:30:22.748ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
133312025-04-30T06:30:22.750ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
133322025-04-30T06:30:22.750ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
133332025-04-30T06:30:22.750ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
133342025-04-30T06:30:22.751ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
133352025-04-30T06:30:22.755ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f53adf8-04ad-483d-ae6d-33b0c1cf44fb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133362025-04-30T06:30:22.755ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
133372025-04-30T06:30:22.756ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
133382025-04-30T06:30:22.756ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
133392025-04-30T06:30:22.761ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d503d23a-8882-47e0-a4ea-942c582d7523", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133402025-04-30T06:30:22.761ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
133412025-04-30T06:30:22.764ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
133422025-04-30T06:30:22.764ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
133432025-04-30T06:30:22.766ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ce7b7f0-7de8-47af-9ac3-f38c7e18d09b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133442025-04-30T06:30:22.766ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
133452025-04-30T06:30:22.769ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14afdf24-5436-4372-8a1d-42ef602082d6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133462025-04-30T06:30:22.769ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
133472025-04-30T06:30:22.770ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c42c5d2-296e-4a97-88c3-039897bc7f1e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133482025-04-30T06:30:22.771ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
133492025-04-30T06:30:22.771ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3079c11f-c26c-4009-8ab3-43ec27830cc5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133502025-04-30T06:30:22.772ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
133512025-04-30T06:30:22.773ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ad01870-c3a9-4dec-b4ad-c78a92718adf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133522025-04-30T06:30:22.773ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
133532025-04-30T06:30:22.774ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53bb0b20-ec96-4251-ac15-9ba82383694f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133542025-04-30T06:30:22.774ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
133552025-04-30T06:30:22.776ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7560d3e0-aa43-4594-b452-bc08f824f6fd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133562025-04-30T06:30:22.776ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
133572025-04-30T06:30:22.779ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "af14b91b-3d46-4a79-878c-053dcbb5c68e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133582025-04-30T06:30:22.779ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
133592025-04-30T06:30:22.781ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fb1bea5-43dc-4574-b923-c473a64851bb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133602025-04-30T06:30:22.781ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
133612025-04-30T06:30:22.783ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "23ff893f-7a64-492f-aada-232faf849ae4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133622025-04-30T06:30:22.783ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
133632025-04-30T06:30:22.784ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133642025-04-30T06:30:22.785ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133652025-04-30T06:30:22.785ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133662025-04-30T06:30:22.785ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
133672025-04-30T06:30:22.786ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133682025-04-30T06:30:22.788ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d658930-1a37-4e40-bad1-ac0ba5bfdeef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133692025-04-30T06:30:22.788ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
133702025-04-30T06:30:22.788ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133712025-04-30T06:30:22.794ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133722025-04-30T06:30:22.794ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133732025-04-30T06:30:22.794ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
133742025-04-30T06:30:22.795ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133752025-04-30T06:30:22.797ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5eb87384-22b3-4f95-a245-b5cc0b898926", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133762025-04-30T06:30:22.797ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
133772025-04-30T06:30:22.797ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133782025-04-30T06:30:22.797ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
133792025-04-30T06:30:22.798ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133802025-04-30T06:30:22.802ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "130ead28-00ba-4748-8d84-cf5e5f673141", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133812025-04-30T06:30:22.802ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
133822025-04-30T06:30:22.804ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "496413db-ff14-4691-8d33-a17c916f798f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133832025-04-30T06:30:22.804ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
133842025-04-30T06:30:22.807ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0130a081-cf51-4e41-b35c-91ab58cda95a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133852025-04-30T06:30:22.808ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
133862025-04-30T06:30:22.811ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "adaba91a-4cd8-4001-a63d-def1b8f9e4f0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133872025-04-30T06:30:22.811ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
133882025-04-30T06:30:22.813ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bb499ef-f86a-4647-9f8a-db2f824f8c12", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133892025-04-30T06:30:22.813ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
133902025-04-30T06:30:22.817ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4dbabb0-5ba8-4994-8229-b6f66aa05f24", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:21 GMT"} })
133912025-04-30T06:30:22.817ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
133922025-04-30T06:30:22.819ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e64b520a-f07f-42b4-ab00-68ec472b1cb0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
133932025-04-30T06:30:22.819ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
133942025-04-30T06:30:22.822ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57cf0743-12e8-4ae0-af09-efe7cb2a102d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
133952025-04-30T06:30:22.822ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
133962025-04-30T06:30:22.826ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f55b3d7-7366-4d65-9b4a-efdbeff957f2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
133972025-04-30T06:30:22.827ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
133982025-04-30T06:30:22.830ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
133992025-04-30T06:30:22.831ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48e5aa5b-b963-484a-b0a7-2ad5360f40fc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
134002025-04-30T06:30:22.831ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
134012025-04-30T06:30:22.833ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b583f968-9640-4f54-8096-eb4743ec7117", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
134022025-04-30T06:30:22.833ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
134032025-04-30T06:30:22.835ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58b54fc9-3b74-4b84-9fd9-2772ad263017", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
134042025-04-30T06:30:22.835ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
134052025-04-30T06:30:22.836ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8679f051-e2b1-44d1-9fe8-12f47513e32b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
134062025-04-30T06:30:22.837ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
134072025-04-30T06:30:22.837ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c2ab737-8655-4abd-8b61-c515f9b785c5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
134082025-04-30T06:30:22.838ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
134092025-04-30T06:30:22.839ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d600361e-c649-450a-9884-9ca6701e63ae", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
134102025-04-30T06:30:22.839ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
134112025-04-30T06:30:22.840ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "351530b3-3c46-4bd3-8b6a-8ac9384d04b8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
134122025-04-30T06:30:22.840ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
134132025-04-30T06:30:22.840ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134142025-04-30T06:30:22.840ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134152025-04-30T06:30:22.840ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134162025-04-30T06:30:22.841ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134172025-04-30T06:30:22.843ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134182025-04-30T06:30:22.847ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
134192025-04-30T06:30:22.847ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
134202025-04-30T06:30:22.847ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134212025-04-30T06:30:22.848ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134222025-04-30T06:30:22.848ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134232025-04-30T06:30:22.848ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134242025-04-30T06:30:22.850ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
134252025-04-30T06:30:22.851ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134262025-04-30T06:30:22.854ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
134272025-04-30T06:30:22.855ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134282025-04-30T06:30:22.855ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134292025-04-30T06:30:22.855ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134302025-04-30T06:30:22.936ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
134312025-04-30T06:30:22.936ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
134322025-04-30T06:30:22.937ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134332025-04-30T06:30:22.937ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134342025-04-30T06:30:22.937ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134352025-04-30T06:30:22.937ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
134362025-04-30T06:30:22.938ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134372025-04-30T06:30:22.939ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:42713 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
134382025-04-30T06:30:22.939ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:42713 req_id = 470c0e45-f7d7-4299-aece-c116d3faf611 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
134392025-04-30T06:30:22.940ZINFOtest_image_promotion (dropshot): request completed latency_us = 555 local_addr = [::1]:60070 method = GET remote_addr = [::1]:42713 req_id = 470c0e45-f7d7-4299-aece-c116d3faf611 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
134402025-04-30T06:30:22.940ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "470c0e45-f7d7-4299-aece-c116d3faf611", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
134412025-04-30T06:30:22.943ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
134422025-04-30T06:30:22.943ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
134432025-04-30T06:30:22.943ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
134442025-04-30T06:30:22.945ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
134452025-04-30T06:30:22.945ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134462025-04-30T06:30:22.945ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134472025-04-30T06:30:22.945ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134482025-04-30T06:30:22.946ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134492025-04-30T06:30:22.946ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134502025-04-30T06:30:22.951ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134512025-04-30T06:30:22.951ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134522025-04-30T06:30:22.951ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
134532025-04-30T06:30:22.951ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134542025-04-30T06:30:22.953ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134552025-04-30T06:30:22.958ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
134562025-04-30T06:30:22.959ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134572025-04-30T06:30:22.959ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134582025-04-30T06:30:22.959ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134592025-04-30T06:30:22.961ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134602025-04-30T06:30:22.961ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
134612025-04-30T06:30:22.962ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134622025-04-30T06:30:23.007ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
134632025-04-30T06:30:23.007ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
134642025-04-30T06:30:23.007ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 378.148236ms iteration = 49 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
134652025-04-30T06:30:23.007ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 50 reason = Signaled
134662025-04-30T06:30:23.007ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
134672025-04-30T06:30:23.007ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134682025-04-30T06:30:23.007ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134692025-04-30T06:30:23.007ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134702025-04-30T06:30:23.008ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134712025-04-30T06:30:23.011ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
134722025-04-30T06:30:23.012ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134732025-04-30T06:30:23.017ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
134742025-04-30T06:30:23.017ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134752025-04-30T06:30:23.017ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134762025-04-30T06:30:23.017ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134772025-04-30T06:30:23.021ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
134782025-04-30T06:30:23.021ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
134792025-04-30T06:30:23.022ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 25 bf 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 %............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50453 req_id = d2414f7c-f4d9-43fc-963b-b936e1ce8155
134802025-04-30T06:30:23.022ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9663,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50453 req_id = d2414f7c-f4d9-43fc-963b-b936e1ce8155
134812025-04-30T06:30:23.022ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
134822025-04-30T06:30:23.022ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
134832025-04-30T06:30:23.022ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50453 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d2414f7c-f4d9-43fc-963b-b936e1ce8155
134842025-04-30T06:30:23.023ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3487, tv_nsec: 821852909 } })
134852025-04-30T06:30:23.023ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
134862025-04-30T06:30:23.024ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e6 91 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52574 req_id = 57a824b2-41d5-49e1-86d7-046a8399f151
134872025-04-30T06:30:23.024ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59025,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52574 req_id = 57a824b2-41d5-49e1-86d7-046a8399f151
134882025-04-30T06:30:23.025ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
134892025-04-30T06:30:23.025ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52574 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 57a824b2-41d5-49e1-86d7-046a8399f151
134902025-04-30T06:30:23.087ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
134912025-04-30T06:30:23.087ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134922025-04-30T06:30:23.087ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134932025-04-30T06:30:23.087ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134942025-04-30T06:30:23.087ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
134952025-04-30T06:30:23.088ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134962025-04-30T06:30:23.088ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 83 97 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57454 req_id = cbd7a630-499f-402c-8a43-121877ee7dfa
134972025-04-30T06:30:23.088ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33687,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57454 req_id = cbd7a630-499f-402c-8a43-121877ee7dfa
134982025-04-30T06:30:23.089ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
134992025-04-30T06:30:23.089ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33687, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57454 req_id = cbd7a630-499f-402c-8a43-121877ee7dfa
135002025-04-30T06:30:23.089ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
135012025-04-30T06:30:23.089ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
135022025-04-30T06:30:23.089ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
135032025-04-30T06:30:23.090ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135042025-04-30T06:30:23.095ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135052025-04-30T06:30:23.095ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135062025-04-30T06:30:23.095ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
135072025-04-30T06:30:23.095ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135082025-04-30T06:30:23.096ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0aa4a825-114d-4083-b3f3-d89e2b8e3628", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135092025-04-30T06:30:23.096ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
135102025-04-30T06:30:23.098ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6787badf-7dd4-4e65-bd7d-e85cab832df2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135112025-04-30T06:30:23.098ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
135122025-04-30T06:30:23.098ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135132025-04-30T06:30:23.098ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
135142025-04-30T06:30:23.099ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135152025-04-30T06:30:23.101ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0837f5dc-0add-4fd2-86bc-cd0218f17d11", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135162025-04-30T06:30:23.101ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
135172025-04-30T06:30:23.104ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d75825d-0503-4d72-83aa-574bf8115c49", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135182025-04-30T06:30:23.104ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
135192025-04-30T06:30:23.111ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ace9a188-77cd-4909-9176-5b034a5f5be5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135202025-04-30T06:30:23.111ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
135212025-04-30T06:30:23.113ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f483979a-e798-4e1f-b8b9-9f4cd932a5c0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135222025-04-30T06:30:23.114ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
135232025-04-30T06:30:23.116ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "337f7453-87ea-441a-afe7-41af1ad09caf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135242025-04-30T06:30:23.116ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
135252025-04-30T06:30:23.119ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21d7889c-e657-4f36-ac63-5978522e9c66", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135262025-04-30T06:30:23.119ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
135272025-04-30T06:30:23.120ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ceaace0-e799-4696-b788-d80afddd20a1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135282025-04-30T06:30:23.120ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
135292025-04-30T06:30:23.121ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05a27128-acd3-4c62-85b4-b22950bb2d9a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135302025-04-30T06:30:23.121ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
135312025-04-30T06:30:23.122ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2606f3ab-ab92-4c0a-ae2c-6398be44f257", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135322025-04-30T06:30:23.122ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
135332025-04-30T06:30:23.124ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e6e4d43-ff65-4d72-a9d0-1b11bd92dd36", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135342025-04-30T06:30:23.124ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
135352025-04-30T06:30:23.126ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6613c5aa-906a-41b5-b2cd-74eef7d3d097", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135362025-04-30T06:30:23.126ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
135372025-04-30T06:30:23.130ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
135382025-04-30T06:30:23.132ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76594b54-4b5b-4554-874c-5d94fbecc13e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135392025-04-30T06:30:23.132ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
135402025-04-30T06:30:23.135ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e747370e-8ed8-4d78-9761-df5129c392ba", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135412025-04-30T06:30:23.135ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
135422025-04-30T06:30:23.137ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7694bbf-f44b-4367-a22f-a2b79f7f337c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135432025-04-30T06:30:23.137ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
135442025-04-30T06:30:23.140ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01773b1e-ba15-40d9-afa2-c5b3fb65e3c3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135452025-04-30T06:30:23.140ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
135462025-04-30T06:30:23.152ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "78870e31-7b5d-4992-b159-579c19ad6fb9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135472025-04-30T06:30:23.152ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
135482025-04-30T06:30:23.156ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbe001a0-0b86-419e-a473-9e4ceae0035f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135492025-04-30T06:30:23.156ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
135502025-04-30T06:30:23.157ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bfda9a1-7d3f-461c-bbd5-5f1b7a7125cd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135512025-04-30T06:30:23.157ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
135522025-04-30T06:30:23.158ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a8f50f6-1411-4918-9e5d-d0c4bfa2ea16", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135532025-04-30T06:30:23.158ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
135542025-04-30T06:30:23.161ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28be815e-4e39-4b53-82c7-a04ce49a3210", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135552025-04-30T06:30:23.161ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
135562025-04-30T06:30:23.163ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28e1720d-3384-4fa6-b5e7-c08a4410d664", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135572025-04-30T06:30:23.163ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
135582025-04-30T06:30:23.166ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a489397e-f22c-4a94-a36f-bd7f4faf32c1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135592025-04-30T06:30:23.166ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
135602025-04-30T06:30:23.170ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a275ce39-1c01-4323-b24b-d690c8dbb40d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135612025-04-30T06:30:23.170ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
135622025-04-30T06:30:23.172ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5742c10e-45e3-48cd-ba91-04e9a5ea17f6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135632025-04-30T06:30:23.172ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
135642025-04-30T06:30:23.174ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50b90634-7d4c-4158-adc4-57da06b2cde2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135652025-04-30T06:30:23.174ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
135662025-04-30T06:30:23.177ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "871dc154-b5ab-4473-b218-b1de14c1011e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135672025-04-30T06:30:23.178ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
135682025-04-30T06:30:23.180ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5a01468-0305-4555-87fe-a2e9990da913", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135692025-04-30T06:30:23.180ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
135702025-04-30T06:30:23.181ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135712025-04-30T06:30:23.182ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135722025-04-30T06:30:23.182ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135732025-04-30T06:30:23.182ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
135742025-04-30T06:30:23.183ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135752025-04-30T06:30:23.184ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce7ca39d-d474-4641-a877-7a773ab576b3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135762025-04-30T06:30:23.184ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
135772025-04-30T06:30:23.185ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5c0a917-6d4a-4a25-a9d2-cb755701eed7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135782025-04-30T06:30:23.185ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
135792025-04-30T06:30:23.186ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135802025-04-30T06:30:23.194ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135812025-04-30T06:30:23.194ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135822025-04-30T06:30:23.194ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
135832025-04-30T06:30:23.194ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135842025-04-30T06:30:23.197ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dff8dbe-a8e6-46af-b80b-8fcc1f51259c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135852025-04-30T06:30:23.197ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
135862025-04-30T06:30:23.197ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135872025-04-30T06:30:23.197ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
135882025-04-30T06:30:23.198ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135892025-04-30T06:30:23.202ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f356357-041d-4317-8762-35dd542d5e23", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135902025-04-30T06:30:23.202ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
135912025-04-30T06:30:23.204ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b964f4cd-3def-4f05-a2ca-07b7f15f8630", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135922025-04-30T06:30:23.204ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
135932025-04-30T06:30:23.207ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c29b2905-8c85-4087-ac45-a49548565f34", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135942025-04-30T06:30:23.208ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
135952025-04-30T06:30:23.215ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "385d04a4-267b-4239-9f6f-d1eb3fd438e5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135962025-04-30T06:30:23.216ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
135972025-04-30T06:30:23.219ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "810d5592-b00d-47b1-8d14-5a9710fe266d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
135982025-04-30T06:30:23.219ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
135992025-04-30T06:30:23.223ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbf7f7ef-20c6-469b-8062-b958b5b52f04", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
136002025-04-30T06:30:23.223ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
136012025-04-30T06:30:23.235ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b807d439-5ec8-4483-bfb5-38126ac0d4f4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
136022025-04-30T06:30:23.235ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
136032025-04-30T06:30:23.238ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb8682bb-7809-4129-962a-338582b891e8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
136042025-04-30T06:30:23.238ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
136052025-04-30T06:30:23.242ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
136062025-04-30T06:30:23.246ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef529158-fdbf-4ce4-8f6e-b53a09fb485c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
136072025-04-30T06:30:23.255ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
136082025-04-30T06:30:23.257ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1e6f1ed-4297-48be-9bc4-d1272e1a5a3a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
136092025-04-30T06:30:23.257ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
136102025-04-30T06:30:23.258ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb62f7da-8f1a-4ada-88ac-52d39700340e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
136112025-04-30T06:30:23.258ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
136122025-04-30T06:30:23.259ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d7f9ade-6619-4a1c-bf42-a7fc0e14a313", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
136132025-04-30T06:30:23.259ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
136142025-04-30T06:30:23.260ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15d8dcd0-78a0-4367-aed2-600547fe9885", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
136152025-04-30T06:30:23.260ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
136162025-04-30T06:30:23.260ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136172025-04-30T06:30:23.261ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136182025-04-30T06:30:23.261ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136192025-04-30T06:30:23.262ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136202025-04-30T06:30:23.267ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136212025-04-30T06:30:23.273ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
136222025-04-30T06:30:23.274ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
136232025-04-30T06:30:23.274ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136242025-04-30T06:30:23.274ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136252025-04-30T06:30:23.274ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136262025-04-30T06:30:23.278ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136272025-04-30T06:30:23.279ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
136282025-04-30T06:30:23.280ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 85 26 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .&..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:57933 req_id = 897a46c1-588c-47fb-86aa-76d49c956ac9
136292025-04-30T06:30:23.280ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34086,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57933 req_id = 897a46c1-588c-47fb-86aa-76d49c956ac9
136302025-04-30T06:30:23.281ZDEBGtest_image_promotion (store): query key key = _dendrite._tcp
136312025-04-30T06:30:23.281ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136322025-04-30T06:30:23.281ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57933 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 897a46c1-588c-47fb-86aa-76d49c956ac9
136332025-04-30T06:30:23.282ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136342025-04-30T06:30:23.299ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
136352025-04-30T06:30:23.299ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136362025-04-30T06:30:23.299ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136372025-04-30T06:30:23.300ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136382025-04-30T06:30:23.302ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
136392025-04-30T06:30:23.303ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136402025-04-30T06:30:23.303ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136412025-04-30T06:30:23.303ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
136422025-04-30T06:30:23.305ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136432025-04-30T06:30:23.305ZDEBGtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3488, tv_nsec: 104813369 } })
136442025-04-30T06:30:23.305ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
136452025-04-30T06:30:23.307ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 01 18 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50337 req_id = 39c034ed-fda3-4032-b736-285ec73913a2
136462025-04-30T06:30:23.307ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 280,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50337 req_id = 39c034ed-fda3-4032-b736-285ec73913a2
136472025-04-30T06:30:23.307ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136482025-04-30T06:30:23.307ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50337 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 39c034ed-fda3-4032-b736-285ec73913a2
136492025-04-30T06:30:23.307ZINFOtest_image_promotion (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
136502025-04-30T06:30:23.356ZINFOtest_image_promotion (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
136512025-04-30T06:30:23.356ZDEBGtest_image_promotion (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
136522025-04-30T06:30:23.410ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
136532025-04-30T06:30:23.410ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
136542025-04-30T06:30:23.410ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136552025-04-30T06:30:23.416ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
136562025-04-30T06:30:23.416ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136572025-04-30T06:30:23.416ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
136582025-04-30T06:30:23.417ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136592025-04-30T06:30:23.417ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:51224 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
136602025-04-30T06:30:23.418ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
136612025-04-30T06:30:23.421ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:51224 req_id = 0f3ca97e-e3db-4d69-97b4-d3f9dbbc3729 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
136622025-04-30T06:30:23.422ZINFOtest_image_promotion (dropshot): request completed latency_us = 639 local_addr = [::1]:60070 method = GET remote_addr = [::1]:51224 req_id = 0f3ca97e-e3db-4d69-97b4-d3f9dbbc3729 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
136632025-04-30T06:30:23.423ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 9c 3c 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .<..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:64292 req_id = 9f9b9e5f-691f-481a-8e16-9dcee19d6a95
136642025-04-30T06:30:23.423ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39996,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64292 req_id = 9f9b9e5f-691f-481a-8e16-9dcee19d6a95
136652025-04-30T06:30:23.423ZDEBGtest_image_promotion (store): query key key = _dendrite._tcp
136662025-04-30T06:30:23.424ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136672025-04-30T06:30:23.424ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64292 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9f9b9e5f-691f-481a-8e16-9dcee19d6a95
136682025-04-30T06:30:23.424ZDEBGtest_image_promotion (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[::1]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7abeabc9-97b5-4942-979a-13caa1e41304", "content-length": "26", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
136692025-04-30T06:30:23.424ZINFOtest_image_promotion (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = ::1
136702025-04-30T06:30:23.426ZINFOtest_image_promotion (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager mappings = {\n Switch0: ::1,\n}
136712025-04-30T06:30:23.476ZTRACtest_image_promotion (BackgroundTasks): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:4676/bfd/peers
136722025-04-30T06:30:23.476ZTRACtest_image_promotion (BackgroundTasks): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:4676/bfd/peers", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
136732025-04-30T06:30:23.476ZERROtest_image_promotion (BackgroundTasks): failed to get bfd sessions from mgd: http://[::1]:4676 background_task = bfd_manager error = Communication Error: error sending request for url (http://[::1]:4676/bfd/peers)
136742025-04-30T06:30:23.477ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 6.984422024s iteration = 2 status = {}
136752025-04-30T06:30:23.478ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f3ca97e-e3db-4d69-97b4-d3f9dbbc3729", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
136762025-04-30T06:30:23.480ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
136772025-04-30T06:30:23.480ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
136782025-04-30T06:30:23.480ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
136792025-04-30T06:30:23.482ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
136802025-04-30T06:30:23.482ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136812025-04-30T06:30:23.482ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136822025-04-30T06:30:23.482ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136832025-04-30T06:30:23.483ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136842025-04-30T06:30:23.484ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136852025-04-30T06:30:23.484ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
136862025-04-30T06:30:23.484ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136872025-04-30T06:30:23.485ZDEBGtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3488, tv_nsec: 284178783 } })
136882025-04-30T06:30:23.485ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
136892025-04-30T06:30:23.486ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ad 20 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e . ..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51877 req_id = a7c1c57b-5351-41e5-a879-373b30af31b8
136902025-04-30T06:30:23.487ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44320,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51877 req_id = a7c1c57b-5351-41e5-a879-373b30af31b8
136912025-04-30T06:30:23.487ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136922025-04-30T06:30:23.488ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51877 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a7c1c57b-5351-41e5-a879-373b30af31b8
136932025-04-30T06:30:23.488ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136942025-04-30T06:30:23.496ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
136952025-04-30T06:30:23.497ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136962025-04-30T06:30:23.497ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136972025-04-30T06:30:23.497ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136982025-04-30T06:30:23.499ZINFOtest_image_promotion (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
136992025-04-30T06:30:23.556ZINFOtest_image_promotion (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
137002025-04-30T06:30:23.556ZDEBGtest_image_promotion (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
137012025-04-30T06:30:23.557ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
137022025-04-30T06:30:23.557ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
137032025-04-30T06:30:23.559ZDEBGtest_image_promotion (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Ok(Response { url: "http://[::1]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c7965f2-4fc1-47dd-a452-a66f67012960", "content-length": "26", "date": "Wed, 30 Apr 2025 06:30:22 GMT"} })
137042025-04-30T06:30:23.559ZINFOtest_image_promotion (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = ::1
137052025-04-30T06:30:23.562ZINFOtest_image_promotion (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager mappings = {\n Switch0: ::1,\n} rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
137062025-04-30T06:30:23.727ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
137072025-04-30T06:30:23.727ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
137082025-04-30T06:30:23.728ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
137092025-04-30T06:30:23.729ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
137102025-04-30T06:30:23.730ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
137112025-04-30T06:30:23.730ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
137122025-04-30T06:30:23.730ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
137132025-04-30T06:30:23.735ZTRACtest_image_promotion (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12224/port/qsfp0/settings?tag=omicron
137142025-04-30T06:30:23.750ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137152025-04-30T06:30:23.750ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137162025-04-30T06:30:23.750ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
137172025-04-30T06:30:23.751ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
137182025-04-30T06:30:23.758ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
137192025-04-30T06:30:23.758ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
137202025-04-30T06:30:23.758ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
137212025-04-30T06:30:23.758ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
137222025-04-30T06:30:23.758ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 751.117643ms iteration = 50 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
137232025-04-30T06:30:23.758ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 51 reason = Signaled
137242025-04-30T06:30:23.758ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
137252025-04-30T06:30:23.758ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137262025-04-30T06:30:23.758ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137272025-04-30T06:30:23.758ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137282025-04-30T06:30:23.759ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137292025-04-30T06:30:23.760ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
137302025-04-30T06:30:23.765ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
137312025-04-30T06:30:23.765ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137322025-04-30T06:30:23.766ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137332025-04-30T06:30:23.770ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
137342025-04-30T06:30:23.770ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137352025-04-30T06:30:23.770ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137362025-04-30T06:30:23.770ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137372025-04-30T06:30:23.774ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
137382025-04-30T06:30:23.774ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
137392025-04-30T06:30:23.775ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 49 fc 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 I............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49223 req_id = f8601d89-252b-431f-a350-494257e5fdec
137402025-04-30T06:30:23.775ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18940,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49223 req_id = f8601d89-252b-431f-a350-494257e5fdec
137412025-04-30T06:30:23.776ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
137422025-04-30T06:30:23.776ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137432025-04-30T06:30:23.776ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49223 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f8601d89-252b-431f-a350-494257e5fdec
137442025-04-30T06:30:23.776ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3488, tv_nsec: 575824859 } })
137452025-04-30T06:30:23.776ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
137462025-04-30T06:30:23.777ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 97 fb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60085 req_id = def96f25-4f6b-4d73-918c-779184b6bc98
137472025-04-30T06:30:23.777ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38907,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60085 req_id = def96f25-4f6b-4d73-918c-779184b6bc98
137482025-04-30T06:30:23.777ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137492025-04-30T06:30:23.777ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60085 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = def96f25-4f6b-4d73-918c-779184b6bc98
137502025-04-30T06:30:23.836ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
137512025-04-30T06:30:23.836ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
137522025-04-30T06:30:23.836ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137532025-04-30T06:30:23.836ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137542025-04-30T06:30:23.836ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
137552025-04-30T06:30:23.837ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137562025-04-30T06:30:23.838ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d6 0d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62512 req_id = e659d07d-39d2-47aa-acaa-dbf43d5dfb1b
137572025-04-30T06:30:23.838ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54797,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62512 req_id = e659d07d-39d2-47aa-acaa-dbf43d5dfb1b
137582025-04-30T06:30:23.838ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
137592025-04-30T06:30:23.838ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54797, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62512 req_id = e659d07d-39d2-47aa-acaa-dbf43d5dfb1b
137602025-04-30T06:30:23.839ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
137612025-04-30T06:30:23.839ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
137622025-04-30T06:30:23.839ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
137632025-04-30T06:30:23.840ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137642025-04-30T06:30:23.849ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
137652025-04-30T06:30:23.849ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137662025-04-30T06:30:23.849ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
137672025-04-30T06:30:23.850ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137682025-04-30T06:30:23.851ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c77263be-93df-4c1d-b733-dc1f6f918c18", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137692025-04-30T06:30:23.852ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
137702025-04-30T06:30:23.853ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
137712025-04-30T06:30:23.855ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137722025-04-30T06:30:23.855ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
137732025-04-30T06:30:23.856ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137742025-04-30T06:30:23.857ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29a1a079-c9ff-4464-8c6f-2244ec4e6818", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137752025-04-30T06:30:23.858ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
137762025-04-30T06:30:23.861ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffd3ac77-0b61-4dd5-85a5-7b4a64273444", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137772025-04-30T06:30:23.861ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
137782025-04-30T06:30:23.864ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1a6b5bf-b3d8-4793-933c-018b5f4287f1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137792025-04-30T06:30:23.865ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
137802025-04-30T06:30:23.868ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ecd2891-31fc-47cc-9ef7-4e2872a6f559", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137812025-04-30T06:30:23.869ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
137822025-04-30T06:30:23.872ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "11297dd6-55e1-4ac4-acfb-40625f07e77f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137832025-04-30T06:30:23.872ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
137842025-04-30T06:30:23.874ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "091f991a-ac8a-4efc-8e1b-83629089a9fc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137852025-04-30T06:30:23.874ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
137862025-04-30T06:30:23.877ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bdb2c27-376c-4854-8577-5a69a5591829", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137872025-04-30T06:30:23.877ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
137882025-04-30T06:30:23.878ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "269f6862-93e7-4b60-b507-ed8042b8c448", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137892025-04-30T06:30:23.878ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
137902025-04-30T06:30:23.880ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a526c18-922a-484a-8e02-8404b4793872", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137912025-04-30T06:30:23.881ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
137922025-04-30T06:30:23.882ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "455fa874-5575-485d-8964-553b81b5d098", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137932025-04-30T06:30:23.883ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
137942025-04-30T06:30:23.884ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d63f5e5-de9c-412b-b6e5-eae37604dd1b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137952025-04-30T06:30:23.884ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
137962025-04-30T06:30:23.889ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3ab52b5-53f5-4ce2-8a41-c4ed911300f4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
137972025-04-30T06:30:23.889ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
137982025-04-30T06:30:23.890ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
137992025-04-30T06:30:23.892ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4de62870-9481-439d-97aa-8cec973ebd3f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138002025-04-30T06:30:23.892ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
138012025-04-30T06:30:23.895ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f97f4bb1-8859-43b8-9d14-35060ef7094f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138022025-04-30T06:30:23.895ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
138032025-04-30T06:30:23.897ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "561e6702-f1db-443f-8b8d-6feb0d7ab488", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138042025-04-30T06:30:23.897ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
138052025-04-30T06:30:23.898ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1830a44-34d2-4365-a38c-d88c2c3275dd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138062025-04-30T06:30:23.898ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
138072025-04-30T06:30:23.900ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50f72605-c4db-4e11-b3b6-38955cc2fd21", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138082025-04-30T06:30:23.901ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
138092025-04-30T06:30:23.903ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53a4b4dd-9b34-4e2c-a8aa-3b94b2bc3be4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138102025-04-30T06:30:23.904ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
138112025-04-30T06:30:23.906ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ab02c2f-9e57-44d8-b655-1375ab9d30b7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138122025-04-30T06:30:23.906ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
138132025-04-30T06:30:23.908ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f09d319d-3144-4fb0-8018-dc22208c1865", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138142025-04-30T06:30:23.908ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
138152025-04-30T06:30:23.910ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d543df84-2353-4700-bafe-94d2f05fd658", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138162025-04-30T06:30:23.910ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
138172025-04-30T06:30:23.911ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5866f523-ab0e-4f72-b5c5-7c6e07a869e1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138182025-04-30T06:30:23.911ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
138192025-04-30T06:30:23.914ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "facbacf0-ba70-4307-9804-b3ac50aec622", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138202025-04-30T06:30:23.914ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
138212025-04-30T06:30:23.918ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4a78202-c0c1-4417-bb1d-3cb18bc66a37", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138222025-04-30T06:30:23.919ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
138232025-04-30T06:30:23.921ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e299303-e59c-494f-b186-d37eabc1c4cb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138242025-04-30T06:30:23.921ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
138252025-04-30T06:30:23.923ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c53da45-31d5-41e8-845c-eec1a1320a73", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138262025-04-30T06:30:23.923ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
138272025-04-30T06:30:23.924ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffbd04bf-117e-41d1-a2a1-5d03ec66f2a1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138282025-04-30T06:30:23.925ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
138292025-04-30T06:30:23.927ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ff43a91-3ada-4098-832b-4256769c2df5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138302025-04-30T06:30:23.927ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
138312025-04-30T06:30:23.930ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7246bb0-a333-46d0-8a09-1074d05597bf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138322025-04-30T06:30:23.930ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
138332025-04-30T06:30:23.931ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dc7743d-6f3b-4d34-a853-7d563755b394", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138342025-04-30T06:30:23.932ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
138352025-04-30T06:30:23.937ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "125b9912-f4a3-414e-b22f-d4d3fb990770", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138362025-04-30T06:30:23.937ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
138372025-04-30T06:30:23.937ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea397528-3ba8-4ac7-b6da-2d10c2ce4390", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138382025-04-30T06:30:23.937ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
138392025-04-30T06:30:23.937ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9352e6df-2b60-4fe4-a77f-41a7a47717fd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138402025-04-30T06:30:23.937ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
138412025-04-30T06:30:23.942ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ecd3c86-a72d-4503-88c6-39e49ae06501", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138422025-04-30T06:30:23.942ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
138432025-04-30T06:30:23.942ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
138442025-04-30T06:30:23.942ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138452025-04-30T06:30:23.942ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138462025-04-30T06:30:23.942ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
138472025-04-30T06:30:23.942ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138482025-04-30T06:30:23.946ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138492025-04-30T06:30:23.959ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
138502025-04-30T06:30:23.959ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138512025-04-30T06:30:23.959ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
138522025-04-30T06:30:23.960ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138532025-04-30T06:30:23.960ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a5ba0d6-a8ae-43b4-be7e-55247250f9f9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138542025-04-30T06:30:23.961ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
138552025-04-30T06:30:23.964ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138562025-04-30T06:30:23.964ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
138572025-04-30T06:30:23.965ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138582025-04-30T06:30:24.008ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a468f9e-9f7c-426c-9348-6d7c0983a0ac", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138592025-04-30T06:30:24.009ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
138602025-04-30T06:30:24.016ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dba2eb7-ee44-4536-a448-2ead11aa5a43", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138612025-04-30T06:30:24.016ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
138622025-04-30T06:30:24.019ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95fd1d60-0c26-47ab-a936-2589b683b5d8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138632025-04-30T06:30:24.019ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
138642025-04-30T06:30:24.022ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17d74d51-e05f-4bf1-a59b-ca17a7236b6c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138652025-04-30T06:30:24.022ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
138662025-04-30T06:30:24.026ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
138672025-04-30T06:30:24.029ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddda9286-8008-40d4-b182-9b30695ddd81", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138682025-04-30T06:30:24.030ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
138692025-04-30T06:30:24.031ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78dc5105-18a5-4339-8402-1e2a9c804099", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138702025-04-30T06:30:24.031ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
138712025-04-30T06:30:24.032ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12036235-cd66-4ae5-b295-7cd467f057b7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138722025-04-30T06:30:24.033ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
138732025-04-30T06:30:24.035ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42414909-404e-44cd-ae6a-d3d7715e9dcb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138742025-04-30T06:30:24.035ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
138752025-04-30T06:30:24.040ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2d1e422-24af-4c1b-8876-dc8d2970c0dc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
138762025-04-30T06:30:24.040ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
138772025-04-30T06:30:24.040ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138782025-04-30T06:30:24.040ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138792025-04-30T06:30:24.040ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138802025-04-30T06:30:24.041ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138812025-04-30T06:30:24.044ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138822025-04-30T06:30:24.052ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
138832025-04-30T06:30:24.052ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
138842025-04-30T06:30:24.053ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138852025-04-30T06:30:24.053ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138862025-04-30T06:30:24.053ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138872025-04-30T06:30:24.053ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138882025-04-30T06:30:24.058ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138892025-04-30T06:30:24.074ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
138902025-04-30T06:30:24.074ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138912025-04-30T06:30:24.074ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138922025-04-30T06:30:24.074ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138932025-04-30T06:30:24.081ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
138942025-04-30T06:30:24.081ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138952025-04-30T06:30:24.081ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138962025-04-30T06:30:24.081ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
138972025-04-30T06:30:24.082ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138982025-04-30T06:30:24.087ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138992025-04-30T06:30:24.106ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
139002025-04-30T06:30:24.106ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139012025-04-30T06:30:24.106ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
139022025-04-30T06:30:24.112ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139032025-04-30T06:30:24.234ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
139042025-04-30T06:30:24.234ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
139052025-04-30T06:30:24.235ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:56514 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
139062025-04-30T06:30:24.235ZTRACtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
139072025-04-30T06:30:24.238ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139082025-04-30T06:30:24.238ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
139092025-04-30T06:30:24.238ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139102025-04-30T06:30:24.242ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:56514 req_id = 71864dad-c287-491c-af28-05837e93ee27 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
139112025-04-30T06:30:24.244ZINFOtest_image_promotion (dropshot): request completed latency_us = 2347 local_addr = [::1]:60070 method = GET remote_addr = [::1]:56514 req_id = 71864dad-c287-491c-af28-05837e93ee27 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
139122025-04-30T06:30:24.244ZTRACtest_image_promotion (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 10 9f 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:62271 req_id = f372f7d2-e917-4c92-af20-1e229ddc6db7
139132025-04-30T06:30:24.244ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4255,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62271 req_id = f372f7d2-e917-4c92-af20-1e229ddc6db7
139142025-04-30T06:30:24.247ZDEBGtest_image_promotion (store): query key key = _dendrite._tcp
139152025-04-30T06:30:24.247ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139162025-04-30T06:30:24.247ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62271 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f372f7d2-e917-4c92-af20-1e229ddc6db7
139172025-04-30T06:30:24.247ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "71864dad-c287-491c-af28-05837e93ee27", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
139182025-04-30T06:30:24.253ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
139192025-04-30T06:30:24.255ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
139202025-04-30T06:30:24.255ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
139212025-04-30T06:30:24.257ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
139222025-04-30T06:30:24.257ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139232025-04-30T06:30:24.257ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139242025-04-30T06:30:24.257ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139252025-04-30T06:30:24.258ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139262025-04-30T06:30:24.259ZDEBGtest_image_promotion (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41733, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3489, tv_nsec: 58635147 } })
139272025-04-30T06:30:24.259ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
139282025-04-30T06:30:24.260ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a0 23 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .#..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49726 req_id = e2f50077-ec88-4fa3-a807-4fb99543aee9
139292025-04-30T06:30:24.260ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40995,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49726 req_id = e2f50077-ec88-4fa3-a807-4fb99543aee9
139302025-04-30T06:30:24.260ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139312025-04-30T06:30:24.261ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49726 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e2f50077-ec88-4fa3-a807-4fb99543aee9
139322025-04-30T06:30:24.262ZINFOtest_image_promotion (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
139332025-04-30T06:30:24.307ZINFOtest_image_promotion (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
139342025-04-30T06:30:24.308ZDEBGtest_image_promotion (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
139352025-04-30T06:30:24.308ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139362025-04-30T06:30:24.311ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
139372025-04-30T06:30:24.312ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139382025-04-30T06:30:24.312ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139392025-04-30T06:30:24.312ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139402025-04-30T06:30:24.328ZDEBGtest_image_promotion (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[::1]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cd4ca22-dcf1-47fa-ae00-c0257e278ad8", "content-length": "26", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
139412025-04-30T06:30:24.328ZINFOtest_image_promotion (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = ::1
139422025-04-30T06:30:24.329ZINFOtest_image_promotion (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector mappings = {\n Switch0: ::1,\n}
139432025-04-30T06:30:24.389ZTRACtest_image_promotion (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12224/rpw/nat/ipv4/gen
139442025-04-30T06:30:24.389ZTRACtest_image_promotion (DpdClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12224/rpw/nat/ipv4/gen", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
139452025-04-30T06:30:24.390ZWARNtest_image_promotion (BackgroundTasks): failed to read generation of dpd background_task = nat_v4_garbage_collector error = Communication Error: error sending request for url (http://[::1]:12224/rpw/nat/ipv4/gen): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146)
139462025-04-30T06:30:24.390ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 7.898445697s iteration = 2 status = {"error":"failed to read generation of dpd: Communication Error: error sending request for url (http://[::1]:12224/rpw/nat/ipv4/gen): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146)"}
139472025-04-30T06:30:24.432ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
139482025-04-30T06:30:24.434ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
139492025-04-30T06:30:24.435ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
139502025-04-30T06:30:24.435ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 676.321005ms iteration = 51 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
139512025-04-30T06:30:24.435ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 52 reason = Signaled
139522025-04-30T06:30:24.435ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
139532025-04-30T06:30:24.435ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139542025-04-30T06:30:24.436ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139552025-04-30T06:30:24.436ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139562025-04-30T06:30:24.438ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139572025-04-30T06:30:24.444ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139582025-04-30T06:30:24.451ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
139592025-04-30T06:30:24.451ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139602025-04-30T06:30:24.451ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139612025-04-30T06:30:24.452ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139622025-04-30T06:30:24.455ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
139632025-04-30T06:30:24.455ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
139642025-04-30T06:30:24.458ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 82 28 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .(..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63158 req_id = d6a46aa6-5376-4590-a64e-9f22ac05814d
139652025-04-30T06:30:24.458ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33320,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63158 req_id = d6a46aa6-5376-4590-a64e-9f22ac05814d
139662025-04-30T06:30:24.458ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
139672025-04-30T06:30:24.458ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139682025-04-30T06:30:24.458ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63158 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d6a46aa6-5376-4590-a64e-9f22ac05814d
139692025-04-30T06:30:24.458ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3489, tv_nsec: 256088910 } })
139702025-04-30T06:30:24.458ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
139712025-04-30T06:30:24.458ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9e bc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60234 req_id = 3c206a3c-7205-4500-a14b-dd5cf8408728
139722025-04-30T06:30:24.458ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40636,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60234 req_id = 3c206a3c-7205-4500-a14b-dd5cf8408728
139732025-04-30T06:30:24.458ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139742025-04-30T06:30:24.458ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60234 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3c206a3c-7205-4500-a14b-dd5cf8408728
139752025-04-30T06:30:24.543ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
139762025-04-30T06:30:24.543ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
139772025-04-30T06:30:24.543ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139782025-04-30T06:30:24.543ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139792025-04-30T06:30:24.543ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
139802025-04-30T06:30:24.544ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139812025-04-30T06:30:24.545ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
139822025-04-30T06:30:24.545ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139832025-04-30T06:30:24.545ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139842025-04-30T06:30:24.546ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139852025-04-30T06:30:24.546ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 7ba177c0-b314-41b8-8bbd-51595d05594b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139862025-04-30T06:30:24.546ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9a 08 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56722 req_id = 789643c8-2298-4927-9a15-df1f64ca5956
139872025-04-30T06:30:24.547ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39432,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56722 req_id = 789643c8-2298-4927-9a15-df1f64ca5956
139882025-04-30T06:30:24.547ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
139892025-04-30T06:30:24.547ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39432, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56722 req_id = 789643c8-2298-4927-9a15-df1f64ca5956
139902025-04-30T06:30:24.547ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 7ba177c0-b314-41b8-8bbd-51595d05594b resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139912025-04-30T06:30:24.547ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 7ba177c0-b314-41b8-8bbd-51595d05594b roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139922025-04-30T06:30:24.548ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 7ba177c0-b314-41b8-8bbd-51595d05594b resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139932025-04-30T06:30:24.551ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
139942025-04-30T06:30:24.551ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
139952025-04-30T06:30:24.553ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
139962025-04-30T06:30:24.553ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139972025-04-30T06:30:24.554ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139982025-04-30T06:30:24.554ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139992025-04-30T06:30:24.554ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
140002025-04-30T06:30:24.555ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140012025-04-30T06:30:24.565ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
140022025-04-30T06:30:24.565ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140032025-04-30T06:30:24.565ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
140042025-04-30T06:30:24.566ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140052025-04-30T06:30:24.567ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 20596 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 7ba177c0-b314-41b8-8bbd-51595d05594b response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
140062025-04-30T06:30:24.568ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ba177c0-b314-41b8-8bbd-51595d05594b", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140072025-04-30T06:30:24.569ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
140082025-04-30T06:30:24.570ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140092025-04-30T06:30:24.570ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
140102025-04-30T06:30:24.571ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140112025-04-30T06:30:24.571ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 96de8d2f-a149-4af9-94ca-a8924f1990fa uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
140122025-04-30T06:30:24.572ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 96de8d2f-a149-4af9-94ca-a8924f1990fa resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
140132025-04-30T06:30:24.572ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 96de8d2f-a149-4af9-94ca-a8924f1990fa roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
140142025-04-30T06:30:24.573ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 96de8d2f-a149-4af9-94ca-a8924f1990fa resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
140152025-04-30T06:30:24.574ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed533149-f958-45ce-936f-d4b478d632dd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140162025-04-30T06:30:24.575ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
140172025-04-30T06:30:24.579ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 7338 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 96de8d2f-a149-4af9-94ca-a8924f1990fa response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
140182025-04-30T06:30:24.579ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "96de8d2f-a149-4af9-94ca-a8924f1990fa", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140192025-04-30T06:30:24.579ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
140202025-04-30T06:30:24.579ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
140212025-04-30T06:30:24.579ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
140222025-04-30T06:30:24.580ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c0133ea-d9e0-4056-931d-2db51995e70f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140232025-04-30T06:30:24.580ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
140242025-04-30T06:30:24.582ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16656a48-6c04-4535-b158-67cbc1e69f79", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140252025-04-30T06:30:24.582ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
140262025-04-30T06:30:24.592ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e2d6d59-dd5c-4651-81a9-c92667729bcc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140272025-04-30T06:30:24.592ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
140282025-04-30T06:30:24.597ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fc86739-f9b0-4552-a97f-ebd594726395", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140292025-04-30T06:30:24.597ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
140302025-04-30T06:30:24.601ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06726234-920c-4305-ae34-aacb0d3ed03c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140312025-04-30T06:30:24.601ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
140322025-04-30T06:30:24.603ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4ad485c-4fa2-451e-b1c6-e41b598d5aa0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140332025-04-30T06:30:24.603ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
140342025-04-30T06:30:24.607ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39d29323-840b-412c-ae19-55fbe893b7e3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140352025-04-30T06:30:24.607ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
140362025-04-30T06:30:24.608ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7712ae3e-0ec2-4030-a441-0acc9d59a80a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140372025-04-30T06:30:24.608ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
140382025-04-30T06:30:24.610ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91187ac3-66c9-4517-9115-b4bd2a0c62b5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140392025-04-30T06:30:24.610ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
140402025-04-30T06:30:24.611ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f95f0f3c-5804-4247-8b62-0bb599185887", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140412025-04-30T06:30:24.612ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
140422025-04-30T06:30:24.613ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db3464aa-ac44-4271-b7e1-8e1a81234739", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140432025-04-30T06:30:24.613ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
140442025-04-30T06:30:24.615ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dedbe428-309b-44f8-9bbb-a1e1e8ba158d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140452025-04-30T06:30:24.615ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
140462025-04-30T06:30:24.624ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dbf68c7-eaea-497b-bbc6-3804bb2a55f8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140472025-04-30T06:30:24.624ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
140482025-04-30T06:30:24.625ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
140492025-04-30T06:30:24.629ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c899457e-179c-449e-8113-0fb0020ce46c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140502025-04-30T06:30:24.629ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
140512025-04-30T06:30:24.633ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb8785b2-a55a-4c01-80ea-4712e6e1bb8a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140522025-04-30T06:30:24.633ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
140532025-04-30T06:30:24.635ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ffe4840-25a4-47c7-b984-74266156b249", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140542025-04-30T06:30:24.635ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
140552025-04-30T06:30:24.638ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9031f39-d274-4666-a9b3-c08ee5fe10c7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140562025-04-30T06:30:24.639ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
140572025-04-30T06:30:24.641ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8155f49-5b3e-49d5-ac6a-186dc52ebeaa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140582025-04-30T06:30:24.641ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
140592025-04-30T06:30:24.642ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f16aa23-9c38-4e23-90e8-914b6efc8436", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140602025-04-30T06:30:24.642ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
140612025-04-30T06:30:24.644ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7c7095c-df1d-4a32-a53e-55594f6f4762", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140622025-04-30T06:30:24.644ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
140632025-04-30T06:30:24.645ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43e82b0a-1745-481b-bfd1-6d6bac714883", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140642025-04-30T06:30:24.645ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
140652025-04-30T06:30:24.647ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de202110-3cdf-41e9-8383-241dbe725311", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140662025-04-30T06:30:24.647ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
140672025-04-30T06:30:24.652ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "482b5c60-e5e1-4de7-b4dd-1e13c63c5beb", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140682025-04-30T06:30:24.652ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
140692025-04-30T06:30:24.657ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d14324bf-0b43-403b-86a7-64e2f366376e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140702025-04-30T06:30:24.657ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
140712025-04-30T06:30:24.661ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1224a66d-a3f4-42e2-8e52-d13a3ecef6cf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140722025-04-30T06:30:24.661ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
140732025-04-30T06:30:24.664ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f47c1dd4-bb83-4dd1-bb7c-f0e479de9c20", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140742025-04-30T06:30:24.664ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
140752025-04-30T06:30:24.666ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "771ab5be-654a-486f-afca-c36b3386da7d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140762025-04-30T06:30:24.666ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
140772025-04-30T06:30:24.669ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2462c7b-726c-4c52-9c6b-ec6fc71d3682", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140782025-04-30T06:30:24.669ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
140792025-04-30T06:30:24.674ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0571c0a-f41f-40af-87f1-7925feeb20e7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140802025-04-30T06:30:24.674ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
140812025-04-30T06:30:24.675ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25fbafae-8036-44af-b770-8e3f74cb62a5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140822025-04-30T06:30:24.676ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
140832025-04-30T06:30:24.676ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
140842025-04-30T06:30:24.676ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140852025-04-30T06:30:24.676ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140862025-04-30T06:30:24.676ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
140872025-04-30T06:30:24.681ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140882025-04-30T06:30:24.685ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1e9f7e1-aba2-4581-bb92-5305f8262521", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140892025-04-30T06:30:24.685ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
140902025-04-30T06:30:24.686ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140912025-04-30T06:30:24.700ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
140922025-04-30T06:30:24.700ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140932025-04-30T06:30:24.700ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
140942025-04-30T06:30:24.700ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140952025-04-30T06:30:24.701ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e0949fc-ad4e-4d7a-8583-cc95b6047354", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140962025-04-30T06:30:24.702ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
140972025-04-30T06:30:24.703ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b51106a0-e4f6-4295-b253-c3b47f7a2b84", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
140982025-04-30T06:30:24.703ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
140992025-04-30T06:30:24.704ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141002025-04-30T06:30:24.704ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
141012025-04-30T06:30:24.704ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141022025-04-30T06:30:24.709ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b834f9d-c67b-4abd-af5b-a0ce50908155", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
141032025-04-30T06:30:24.709ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
141042025-04-30T06:30:24.712ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ef34ae7-48ed-47f1-8927-9adc27464ef0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
141052025-04-30T06:30:24.712ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
141062025-04-30T06:30:24.714ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "015b5a24-73eb-4e8f-b352-0c428029736d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
141072025-04-30T06:30:24.714ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
141082025-04-30T06:30:24.717ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a276753-794f-4677-bf68-c8c12ea41910", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
141092025-04-30T06:30:24.717ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
141102025-04-30T06:30:24.723ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a80a72f0-3ef0-4eb5-b876-e5e2409190db", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
141112025-04-30T06:30:24.723ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
141122025-04-30T06:30:24.727ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e12bd345-4612-42bd-a926-bec0b2f2ea27", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
141132025-04-30T06:30:24.727ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
141142025-04-30T06:30:24.730ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8d88bba-56ac-4995-8171-274e21b9f6cc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
141152025-04-30T06:30:24.730ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
141162025-04-30T06:30:24.731ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fe229e9-da09-470a-ae7f-b4738fccb8fc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
141172025-04-30T06:30:24.731ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
141182025-04-30T06:30:24.733ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f66d206d-ad94-48d9-9f8d-caa003c3e34c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
141192025-04-30T06:30:24.733ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
141202025-04-30T06:30:24.736ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
141212025-04-30T06:30:24.736ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9a3a481-c58f-40be-b606-5a79bd3cf53c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
141222025-04-30T06:30:24.736ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
141232025-04-30T06:30:24.738ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9bbf8ca-7afd-4b36-b888-d0fb6cfa9ac3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:23 GMT"} })
141242025-04-30T06:30:24.738ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
141252025-04-30T06:30:24.738ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141262025-04-30T06:30:24.738ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141272025-04-30T06:30:24.738ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141282025-04-30T06:30:24.739ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141292025-04-30T06:30:24.759ZDEBGtest_image_promotion (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
141302025-04-30T06:30:24.759ZDEBGtest_image_promotion (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
141312025-04-30T06:30:24.759ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141322025-04-30T06:30:24.762ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
141332025-04-30T06:30:24.762ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
141342025-04-30T06:30:24.762ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141352025-04-30T06:30:24.763ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141362025-04-30T06:30:24.763ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141372025-04-30T06:30:24.763ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141382025-04-30T06:30:24.763ZDEBGtest_image_promotion (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
141392025-04-30T06:30:24.764ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
141402025-04-30T06:30:24.768ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
141412025-04-30T06:30:24.768ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
141422025-04-30T06:30:24.770ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141432025-04-30T06:30:24.786ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
141442025-04-30T06:30:24.786ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141452025-04-30T06:30:24.786ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141462025-04-30T06:30:24.789ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141472025-04-30T06:30:24.789ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
141482025-04-30T06:30:24.789ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141492025-04-30T06:30:24.789ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141502025-04-30T06:30:24.789ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
141512025-04-30T06:30:24.789ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141522025-04-30T06:30:24.789ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 1 table_name = measurements_i64
141532025-04-30T06:30:24.789ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
141542025-04-30T06:30:24.859ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
141552025-04-30T06:30:24.859ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
141562025-04-30T06:30:24.859ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141572025-04-30T06:30:24.867ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
141582025-04-30T06:30:24.867ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141592025-04-30T06:30:24.867ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
141602025-04-30T06:30:24.868ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141612025-04-30T06:30:24.868ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:62372 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
141622025-04-30T06:30:24.880ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:62372 req_id = 785bddc2-5c9c-47b9-b6f8-1fd2cdb0466c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
141632025-04-30T06:30:24.881ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
141642025-04-30T06:30:24.881ZINFOtest_image_promotion (dropshot): request completed latency_us = 1064 local_addr = [::1]:60070 method = GET remote_addr = [::1]:62372 req_id = 785bddc2-5c9c-47b9-b6f8-1fd2cdb0466c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
141652025-04-30T06:30:24.882ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "785bddc2-5c9c-47b9-b6f8-1fd2cdb0466c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
141662025-04-30T06:30:24.884ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
141672025-04-30T06:30:24.884ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
141682025-04-30T06:30:24.884ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
141692025-04-30T06:30:24.886ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
141702025-04-30T06:30:24.886ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141712025-04-30T06:30:24.886ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141722025-04-30T06:30:24.886ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141732025-04-30T06:30:24.887ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141742025-04-30T06:30:24.888ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141752025-04-30T06:30:24.888ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
141762025-04-30T06:30:24.889ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141772025-04-30T06:30:24.891ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141782025-04-30T06:30:24.895ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
141792025-04-30T06:30:24.895ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141802025-04-30T06:30:24.895ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141812025-04-30T06:30:24.896ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141822025-04-30T06:30:24.917ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:32824 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
141832025-04-30T06:30:24.918ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:32824 req_id = 46a42cd7-9a2f-4735-bd87-d524739e3081 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
141842025-04-30T06:30:24.918ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 300 local_addr = [::1]:65415 method = GET remote_addr = [::1]:32824 req_id = 46a42cd7-9a2f-4735-bd87-d524739e3081 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
141852025-04-30T06:30:24.928ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
141862025-04-30T06:30:24.928ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
141872025-04-30T06:30:24.928ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 493.30099ms iteration = 52 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
141882025-04-30T06:30:24.928ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 53 reason = Signaled
141892025-04-30T06:30:24.929ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
141902025-04-30T06:30:24.929ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141912025-04-30T06:30:24.929ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141922025-04-30T06:30:24.929ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141932025-04-30T06:30:24.931ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141942025-04-30T06:30:24.931ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
141952025-04-30T06:30:24.934ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141962025-04-30T06:30:24.938ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
141972025-04-30T06:30:24.938ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141982025-04-30T06:30:24.938ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141992025-04-30T06:30:24.939ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
142002025-04-30T06:30:24.942ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
142012025-04-30T06:30:24.943ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
142022025-04-30T06:30:24.944ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 26 c3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 &............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52394 req_id = 35debfb5-5ee3-490e-b255-23190c494c4d
142032025-04-30T06:30:24.944ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9923,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52394 req_id = 35debfb5-5ee3-490e-b255-23190c494c4d
142042025-04-30T06:30:24.944ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
142052025-04-30T06:30:24.945ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
142062025-04-30T06:30:24.945ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52394 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 35debfb5-5ee3-490e-b255-23190c494c4d
142072025-04-30T06:30:24.945ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3489, tv_nsec: 744561510 } })
142082025-04-30T06:30:24.945ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
142092025-04-30T06:30:24.946ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 64 ef 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e d...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53739 req_id = 4d8f6b04-2f46-462c-be11-fd141b9ccd28
142102025-04-30T06:30:24.946ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25839,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53739 req_id = 4d8f6b04-2f46-462c-be11-fd141b9ccd28
142112025-04-30T06:30:24.946ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
142122025-04-30T06:30:24.946ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53739 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4d8f6b04-2f46-462c-be11-fd141b9ccd28
142132025-04-30T06:30:25.017ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
142142025-04-30T06:30:25.018ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
142152025-04-30T06:30:25.018ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
142162025-04-30T06:30:25.018ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142172025-04-30T06:30:25.018ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
142182025-04-30T06:30:25.019ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142192025-04-30T06:30:25.021ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 52 cd 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c R............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53019 req_id = 7fdfb01a-5a32-4036-aece-10cb4b5d1f02
142202025-04-30T06:30:25.022ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21197,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53019 req_id = 7fdfb01a-5a32-4036-aece-10cb4b5d1f02
142212025-04-30T06:30:25.022ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
142222025-04-30T06:30:25.022ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21197, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53019 req_id = 7fdfb01a-5a32-4036-aece-10cb4b5d1f02
142232025-04-30T06:30:25.024ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
142242025-04-30T06:30:25.024ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
142252025-04-30T06:30:25.024ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
142262025-04-30T06:30:25.025ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
142272025-04-30T06:30:25.037ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
142282025-04-30T06:30:25.037ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142292025-04-30T06:30:25.037ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
142302025-04-30T06:30:25.038ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142312025-04-30T06:30:25.047ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142322025-04-30T06:30:25.047ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
142332025-04-30T06:30:25.048ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142342025-04-30T06:30:25.057ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f67540b-0be2-4914-b31b-a711757bb2eb", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142352025-04-30T06:30:25.058ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
142362025-04-30T06:30:25.074ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a630c8c-7324-423e-8a92-c6974c863727", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142372025-04-30T06:30:25.074ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
142382025-04-30T06:30:25.085ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d8f8c5c-d0c3-404e-8bcb-c3c242510cff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142392025-04-30T06:30:25.085ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
142402025-04-30T06:30:25.087ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2523db5-3cfb-43da-b955-84962978573c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142412025-04-30T06:30:25.087ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
142422025-04-30T06:30:25.092ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39eb8c76-5a71-4fcb-9d4e-785059985ddf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142432025-04-30T06:30:25.092ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
142442025-04-30T06:30:25.099ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "771ae0a5-5889-4c4a-98f8-552e7f00e12c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142452025-04-30T06:30:25.100ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
142462025-04-30T06:30:25.100ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
142472025-04-30T06:30:25.103ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac9cf1ce-27e7-4a2f-8769-4ae8aa14c440", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142482025-04-30T06:30:25.103ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
142492025-04-30T06:30:25.116ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fc8fb82-4c75-4392-8a73-ced76e4a6951", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142502025-04-30T06:30:25.117ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
142512025-04-30T06:30:25.117ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "380671ae-9675-44e9-8e93-22e3969b8c98", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142522025-04-30T06:30:25.118ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
142532025-04-30T06:30:25.119ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "abd36c54-8748-4eed-a92e-b15acecbec3d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142542025-04-30T06:30:25.119ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
142552025-04-30T06:30:25.120ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d705b51-f29b-42f4-9d61-a93154faf884", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142562025-04-30T06:30:25.120ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
142572025-04-30T06:30:25.121ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22041dc5-d639-4e28-b54e-736eb696c88c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142582025-04-30T06:30:25.121ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
142592025-04-30T06:30:25.123ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7aa29f1c-cf8b-4c27-a3c3-7bd4d292d007", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142602025-04-30T06:30:25.123ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
142612025-04-30T06:30:25.126ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3406b3f-aad7-4551-b049-c6b79e42d006", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142622025-04-30T06:30:25.126ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
142632025-04-30T06:30:25.128ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c5f8b15-99a3-432f-82db-0e779d70868e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142642025-04-30T06:30:25.128ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
142652025-04-30T06:30:25.130ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9990839b-4499-4def-95a4-9c0a9803cc67", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142662025-04-30T06:30:25.130ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
142672025-04-30T06:30:25.132ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "702c53db-354f-4207-ab30-ca387c554282", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142682025-04-30T06:30:25.132ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
142692025-04-30T06:30:25.135ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25ae26e5-bf3b-447d-a8e1-987e9e24999d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142702025-04-30T06:30:25.135ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
142712025-04-30T06:30:25.137ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5ef558d-a591-48ed-97d3-882123fee22f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142722025-04-30T06:30:25.137ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
142732025-04-30T06:30:25.138ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd7950dd-4956-4b03-8854-c3a4de97ae79", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142742025-04-30T06:30:25.139ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
142752025-04-30T06:30:25.140ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bd98931-170f-40bc-8cb9-321aa6de8afb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142762025-04-30T06:30:25.140ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
142772025-04-30T06:30:25.142ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f3f1af6-0051-43d5-9475-3a8d7238c544", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142782025-04-30T06:30:25.142ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
142792025-04-30T06:30:25.143ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a78987b2-9c38-4b7c-b8ed-0905cf7f8cf5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142802025-04-30T06:30:25.143ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
142812025-04-30T06:30:25.145ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "600a5244-208d-4a72-addd-4ef4067d93ac", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142822025-04-30T06:30:25.145ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
142832025-04-30T06:30:25.147ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "72486cdf-aa2a-4757-ad49-63433fbd1f3a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142842025-04-30T06:30:25.147ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
142852025-04-30T06:30:25.150ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8edbd607-b8a5-4493-901b-02c8bd492c0f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142862025-04-30T06:30:25.150ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
142872025-04-30T06:30:25.151ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
142882025-04-30T06:30:25.151ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
142892025-04-30T06:30:25.151ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142902025-04-30T06:30:25.151ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
142912025-04-30T06:30:25.153ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142922025-04-30T06:30:25.155ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5715a2cd-49e5-4846-b0d4-f9b179e10f20", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
142932025-04-30T06:30:25.155ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
142942025-04-30T06:30:25.156ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
142952025-04-30T06:30:25.166ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
142962025-04-30T06:30:25.166ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142972025-04-30T06:30:25.166ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
142982025-04-30T06:30:25.167ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142992025-04-30T06:30:25.171ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143002025-04-30T06:30:25.171ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
143012025-04-30T06:30:25.172ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143022025-04-30T06:30:25.174ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35e17157-2c9a-4c33-936d-e4596e4d059a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143032025-04-30T06:30:25.174ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
143042025-04-30T06:30:25.180ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06abe3e7-0454-49b0-88c4-3d77493e5eae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143052025-04-30T06:30:25.180ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
143062025-04-30T06:30:25.184ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d56f1365-2f52-4804-a5b1-622e22bc33a5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143072025-04-30T06:30:25.185ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
143082025-04-30T06:30:25.187ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fb0e82e-e483-4fac-a876-aa57d3613e73", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143092025-04-30T06:30:25.187ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
143102025-04-30T06:30:25.190ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eccb0370-2f30-4c7d-8474-f338d4b5a482", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143112025-04-30T06:30:25.190ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
143122025-04-30T06:30:25.192ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74bdf227-31ab-4fd6-aa64-1770d7672d5d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143132025-04-30T06:30:25.193ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
143142025-04-30T06:30:25.196ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ed258cb-7c76-4706-9992-3f9a0f79e7e9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143152025-04-30T06:30:25.196ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
143162025-04-30T06:30:25.200ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02648366-dc36-4e72-a486-20af83257110", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143172025-04-30T06:30:25.200ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
143182025-04-30T06:30:25.203ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba5fa03f-b261-41f2-abbe-eba00b0f3771", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143192025-04-30T06:30:25.203ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
143202025-04-30T06:30:25.208ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
143212025-04-30T06:30:25.212ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ac39c19-0c5c-4bf3-96da-c50a97c63e19", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143222025-04-30T06:30:25.213ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
143232025-04-30T06:30:25.216ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fce5c50-664e-48ec-8fdd-cb9c3db60b7b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143242025-04-30T06:30:25.216ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
143252025-04-30T06:30:25.219ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e23c556e-4af3-4c18-bc15-14e1c4344112", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143262025-04-30T06:30:25.219ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
143272025-04-30T06:30:25.223ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec3ac8be-3af0-4f21-a969-8c864002180d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143282025-04-30T06:30:25.223ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
143292025-04-30T06:30:25.225ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06478cfe-90cf-46da-8c36-e104f655b733", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143302025-04-30T06:30:25.225ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
143312025-04-30T06:30:25.226ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f69ba57-7c4c-42a0-845c-af34a40270d7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143322025-04-30T06:30:25.226ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
143332025-04-30T06:30:25.227ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4238cb67-7c81-40ce-b7f6-c5d52d73f4a9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143342025-04-30T06:30:25.227ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
143352025-04-30T06:30:25.228ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80d99631-cbfb-4439-945e-a27afe6485ad", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143362025-04-30T06:30:25.228ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
143372025-04-30T06:30:25.229ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb8cac5a-f5dd-47e6-bc25-38c4c78c78f9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143382025-04-30T06:30:25.230ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
143392025-04-30T06:30:25.230ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143402025-04-30T06:30:25.230ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143412025-04-30T06:30:25.230ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143422025-04-30T06:30:25.231ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143432025-04-30T06:30:25.235ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143442025-04-30T06:30:25.240ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
143452025-04-30T06:30:25.240ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
143462025-04-30T06:30:25.241ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143472025-04-30T06:30:25.241ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143482025-04-30T06:30:25.241ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143492025-04-30T06:30:25.241ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143502025-04-30T06:30:25.246ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143512025-04-30T06:30:25.254ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
143522025-04-30T06:30:25.254ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143532025-04-30T06:30:25.254ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143542025-04-30T06:30:25.255ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143552025-04-30T06:30:25.260ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
143562025-04-30T06:30:25.260ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143572025-04-30T06:30:25.260ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143582025-04-30T06:30:25.260ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
143592025-04-30T06:30:25.262ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143602025-04-30T06:30:25.332ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
143612025-04-30T06:30:25.332ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
143622025-04-30T06:30:25.333ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:34535 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143632025-04-30T06:30:25.334ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:34535 req_id = 2cbc0129-d449-4160-833b-c348238ddeaa sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
143642025-04-30T06:30:25.335ZINFOtest_image_promotion (dropshot): request completed latency_us = 602 local_addr = [::1]:60070 method = GET remote_addr = [::1]:34535 req_id = 2cbc0129-d449-4160-833b-c348238ddeaa response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
143652025-04-30T06:30:25.335ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143662025-04-30T06:30:25.341ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
143672025-04-30T06:30:25.341ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143682025-04-30T06:30:25.341ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
143692025-04-30T06:30:25.341ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143702025-04-30T06:30:25.342ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cbc0129-d449-4160-833b-c348238ddeaa", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:24 GMT"} })
143712025-04-30T06:30:25.346ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
143722025-04-30T06:30:25.346ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
143732025-04-30T06:30:25.346ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
143742025-04-30T06:30:25.348ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
143752025-04-30T06:30:25.348ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143762025-04-30T06:30:25.348ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143772025-04-30T06:30:25.348ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143782025-04-30T06:30:25.348ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143792025-04-30T06:30:25.351ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143802025-04-30T06:30:25.352ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
143812025-04-30T06:30:25.352ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143822025-04-30T06:30:25.354ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143832025-04-30T06:30:25.359ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
143842025-04-30T06:30:25.359ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143852025-04-30T06:30:25.359ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143862025-04-30T06:30:25.360ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143872025-04-30T06:30:25.854ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:46796 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143882025-04-30T06:30:25.854ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:42847 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143892025-04-30T06:30:25.854ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:42860 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143902025-04-30T06:30:25.854ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:47477 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143912025-04-30T06:30:25.854ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:54654 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143922025-04-30T06:30:25.854ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:50359 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143932025-04-30T06:30:25.854ZINFOtest_image_promotion (dropshot): accepted connection kind = pantry local_addr = [::1]:65415 remote_addr = [::1]:59754 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143942025-04-30T06:30:25.854ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143952025-04-30T06:30:25.854ZTRACtest_image_promotion (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
143962025-04-30T06:30:25.855ZTRACtest_image_promotion (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
143972025-04-30T06:30:25.855ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143982025-04-30T06:30:25.857ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:46796 req_id = 29d33c61-7884-4699-9882-db69467a7ab8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143992025-04-30T06:30:25.857ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:42847 req_id = 9f837df3-fe9a-4aab-ab60-c282b692abae sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144002025-04-30T06:30:25.857ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:42860 req_id = 442116ce-63b7-4374-b226-7fd0f32cca57 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144012025-04-30T06:30:25.858ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:47477 req_id = 4e4e6b49-4d7a-4af5-9a98-79ce8a9124c1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144022025-04-30T06:30:25.858ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:54654 req_id = 43ec11a2-9ae8-4078-8987-385211021677 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144032025-04-30T06:30:25.858ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:50359 req_id = 08057762-e056-4568-a95a-1c34b3b7f43d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144042025-04-30T06:30:25.858ZTRACtest_image_promotion (dropshot): incoming request kind = pantry local_addr = [::1]:65415 method = GET remote_addr = [::1]:59754 req_id = 41079f11-55b0-4080-9e27-9f5d4bef5c4c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144052025-04-30T06:30:25.860ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 2870 local_addr = [::1]:65415 method = GET remote_addr = [::1]:46796 req_id = 29d33c61-7884-4699-9882-db69467a7ab8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144062025-04-30T06:30:25.860ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 3016 local_addr = [::1]:65415 method = GET remote_addr = [::1]:42847 req_id = 9f837df3-fe9a-4aab-ab60-c282b692abae response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144072025-04-30T06:30:25.860ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 2983 local_addr = [::1]:65415 method = GET remote_addr = [::1]:42860 req_id = 442116ce-63b7-4374-b226-7fd0f32cca57 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144082025-04-30T06:30:25.861ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 3042 local_addr = [::1]:65415 method = GET remote_addr = [::1]:47477 req_id = 4e4e6b49-4d7a-4af5-9a98-79ce8a9124c1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144092025-04-30T06:30:25.861ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 2989 local_addr = [::1]:65415 method = GET remote_addr = [::1]:54654 req_id = 43ec11a2-9ae8-4078-8987-385211021677 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144102025-04-30T06:30:25.861ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 3649 local_addr = [::1]:65415 method = GET remote_addr = [::1]:50359 req_id = 08057762-e056-4568-a95a-1c34b3b7f43d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144112025-04-30T06:30:25.861ZINFOtest_image_promotion (dropshot): request completed kind = pantry latency_us = 3597 local_addr = [::1]:65415 method = GET remote_addr = [::1]:59754 req_id = 41079f11-55b0-4080-9e27-9f5d4bef5c4c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
144122025-04-30T06:30:25.865ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144132025-04-30T06:30:25.865ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144142025-04-30T06:30:25.865ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144152025-04-30T06:30:25.865ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144162025-04-30T06:30:25.883ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
144172025-04-30T06:30:25.884ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
144182025-04-30T06:30:25.884ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
144192025-04-30T06:30:25.884ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 955.707285ms iteration = 53 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
144202025-04-30T06:30:25.884ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 54 reason = Signaled
144212025-04-30T06:30:25.884ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
144222025-04-30T06:30:25.884ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
144232025-04-30T06:30:25.884ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
144242025-04-30T06:30:25.884ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
144252025-04-30T06:30:25.885ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
144262025-04-30T06:30:25.887ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
144272025-04-30T06:30:25.895ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
144282025-04-30T06:30:25.895ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
144292025-04-30T06:30:25.895ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
144302025-04-30T06:30:25.896ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
144312025-04-30T06:30:25.900ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
144322025-04-30T06:30:25.900ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
144332025-04-30T06:30:25.901ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6c 2c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 l,..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59093 req_id = aa7aab57-7f20-41e3-9b66-7fde0eb6d418
144342025-04-30T06:30:25.901ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27692,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59093 req_id = aa7aab57-7f20-41e3-9b66-7fde0eb6d418
144352025-04-30T06:30:25.902ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
144362025-04-30T06:30:25.902ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
144372025-04-30T06:30:25.902ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59093 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = aa7aab57-7f20-41e3-9b66-7fde0eb6d418
144382025-04-30T06:30:25.903ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3490, tv_nsec: 702141075 } })
144392025-04-30T06:30:25.903ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
144402025-04-30T06:30:25.903ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fa d7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53835 req_id = 4e377e07-7f24-48b9-8592-697b1d717016
144412025-04-30T06:30:25.904ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64215,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53835 req_id = 4e377e07-7f24-48b9-8592-697b1d717016
144422025-04-30T06:30:25.904ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
144432025-04-30T06:30:25.904ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53835 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4e377e07-7f24-48b9-8592-697b1d717016
144442025-04-30T06:30:25.977ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
144452025-04-30T06:30:25.977ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
144462025-04-30T06:30:25.978ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
144472025-04-30T06:30:25.978ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144482025-04-30T06:30:25.978ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
144492025-04-30T06:30:25.980ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144502025-04-30T06:30:25.983ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 86 d0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65155 req_id = d036c470-5b1d-4999-857a-6a677d69e4d8
144512025-04-30T06:30:25.983ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34512,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65155 req_id = d036c470-5b1d-4999-857a-6a677d69e4d8
144522025-04-30T06:30:25.983ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
144532025-04-30T06:30:25.984ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34512, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65155 req_id = d036c470-5b1d-4999-857a-6a677d69e4d8
144542025-04-30T06:30:25.986ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
144552025-04-30T06:30:25.986ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
144562025-04-30T06:30:25.986ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
144572025-04-30T06:30:25.987ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
144582025-04-30T06:30:25.994ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
144592025-04-30T06:30:25.994ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144602025-04-30T06:30:25.994ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
144612025-04-30T06:30:25.995ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144622025-04-30T06:30:26.002ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144632025-04-30T06:30:26.002ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
144642025-04-30T06:30:26.004ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144652025-04-30T06:30:26.014ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceab4842-0dec-43b9-ac50-a677bae61408", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144662025-04-30T06:30:26.014ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
144672025-04-30T06:30:26.030ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68aa94ca-b36c-464e-8f89-6eb847f3ed88", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144682025-04-30T06:30:26.030ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
144692025-04-30T06:30:26.046ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88be92eb-f8bd-42ce-be10-9550f8ba53c4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144702025-04-30T06:30:26.046ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
144712025-04-30T06:30:26.052ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "952d5ff4-854f-4a14-b36e-3d04e24b5b39", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144722025-04-30T06:30:26.053ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
144732025-04-30T06:30:26.053ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
144742025-04-30T06:30:26.055ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9149e09-12c7-4789-99b0-64ee873dcf53", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144752025-04-30T06:30:26.056ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
144762025-04-30T06:30:26.058ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b719339c-4274-45fc-8769-f2bb379c9d3e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144772025-04-30T06:30:26.058ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
144782025-04-30T06:30:26.060ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00925b3f-318e-481d-a45c-1c7f3b9245e1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144792025-04-30T06:30:26.060ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
144802025-04-30T06:30:26.062ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c57eed8-0ae2-4675-a066-63afd09c8e47", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144812025-04-30T06:30:26.062ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
144822025-04-30T06:30:26.064ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1fbf7d3-4af8-445d-b956-b32c072789ae", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144832025-04-30T06:30:26.064ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
144842025-04-30T06:30:26.066ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c29b63f6-4f9a-413f-ba1e-cc925d73f301", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144852025-04-30T06:30:26.067ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
144862025-04-30T06:30:26.068ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc05154c-4b71-4a8b-b4c3-04b3ed21bfe5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144872025-04-30T06:30:26.068ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
144882025-04-30T06:30:26.069ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e9bcbd4-658a-41d1-8a00-144239fb873c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144892025-04-30T06:30:26.069ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
144902025-04-30T06:30:26.071ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73bd8ea8-0eb7-478c-89c6-04d3e02c1d3f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144912025-04-30T06:30:26.071ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
144922025-04-30T06:30:26.073ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e51293aa-795e-4554-ac19-984e743d14f5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144932025-04-30T06:30:26.074ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
144942025-04-30T06:30:26.076ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d925261-231b-4bea-9425-715a3f01f049", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144952025-04-30T06:30:26.076ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
144962025-04-30T06:30:26.078ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b9305f7-5547-48e3-b327-3ce7f8d04bd7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144972025-04-30T06:30:26.078ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
144982025-04-30T06:30:26.080ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "839c4cf1-391c-4d3a-b41a-4fb59da66cc0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
144992025-04-30T06:30:26.080ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
145002025-04-30T06:30:26.084ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05ca6f09-5ba5-4bbe-a86f-2d1ce0816f68", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145012025-04-30T06:30:26.084ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
145022025-04-30T06:30:26.086ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f4bccc2-5fc7-4e3d-b81f-8e882fa85c08", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145032025-04-30T06:30:26.086ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
145042025-04-30T06:30:26.089ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd2dcd2f-080f-4ac5-9888-b9e6b2fc2395", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145052025-04-30T06:30:26.089ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
145062025-04-30T06:30:26.090ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be438891-3752-4ad4-90c8-38b4a149e7fd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145072025-04-30T06:30:26.090ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
145082025-04-30T06:30:26.091ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5378760a-1597-43fb-8bb3-580f1010e0b7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145092025-04-30T06:30:26.091ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
145102025-04-30T06:30:26.092ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "878a25fe-feae-4d6b-9b55-4a08bb3a60e2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145112025-04-30T06:30:26.092ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
145122025-04-30T06:30:26.094ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec33af19-a681-4863-8c2c-f50da294664a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145132025-04-30T06:30:26.094ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
145142025-04-30T06:30:26.100ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad87b1cd-c19d-490a-9936-b5648ab38173", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145152025-04-30T06:30:26.101ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
145162025-04-30T06:30:26.105ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
145172025-04-30T06:30:26.105ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145182025-04-30T06:30:26.105ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145192025-04-30T06:30:26.105ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
145202025-04-30T06:30:26.107ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145212025-04-30T06:30:26.109ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e416ff0-a1d2-4deb-bdb6-a18ec7784265", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145222025-04-30T06:30:26.110ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
145232025-04-30T06:30:26.110ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145242025-04-30T06:30:26.116ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
145252025-04-30T06:30:26.116ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145262025-04-30T06:30:26.116ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
145272025-04-30T06:30:26.117ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145282025-04-30T06:30:26.119ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145292025-04-30T06:30:26.120ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
145302025-04-30T06:30:26.120ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145312025-04-30T06:30:26.136ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "757ee88e-9e79-4039-bfc5-c13d12c02fe6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145322025-04-30T06:30:26.137ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
145332025-04-30T06:30:26.140ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28846182-8382-4588-a38f-977d3db1392d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145342025-04-30T06:30:26.141ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
145352025-04-30T06:30:26.144ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4df3b40c-94ac-4584-868a-8a488141c6ad", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145362025-04-30T06:30:26.144ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
145372025-04-30T06:30:26.152ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9ddbd44-942b-445e-9401-e6dd3b18ffdc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145382025-04-30T06:30:26.152ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
145392025-04-30T06:30:26.153ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21e6d48e-45c0-4680-8463-f2099ad46340", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145402025-04-30T06:30:26.154ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
145412025-04-30T06:30:26.155ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6a99dfc-37be-4e55-a08e-b5e2f4f8653f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145422025-04-30T06:30:26.155ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
145432025-04-30T06:30:26.157ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce57b2db-82df-424a-9c63-e34ec55eebb1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145442025-04-30T06:30:26.158ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
145452025-04-30T06:30:26.160ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad2cafa8-ac0d-408b-98e2-492c465cc45f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145462025-04-30T06:30:26.160ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
145472025-04-30T06:30:26.162ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d74dc05-ba89-4e93-b6c2-b7ba831d03b0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145482025-04-30T06:30:26.163ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
145492025-04-30T06:30:26.179ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
145502025-04-30T06:30:26.180ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f90a6cc6-afac-42c1-86be-ed334573c14f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145512025-04-30T06:30:26.180ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
145522025-04-30T06:30:26.185ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f949731c-e0cb-4705-bab7-6959b03a5070", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145532025-04-30T06:30:26.185ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
145542025-04-30T06:30:26.188ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc7a5d5d-67cd-4267-b273-1df5bdadada5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145552025-04-30T06:30:26.188ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
145562025-04-30T06:30:26.191ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d6d5555-9399-41e5-8b64-4ef2be92c4d9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145572025-04-30T06:30:26.191ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
145582025-04-30T06:30:26.193ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f794eb0-7842-42bc-bc90-3bcb499b2046", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145592025-04-30T06:30:26.194ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
145602025-04-30T06:30:26.196ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e12a61d-4e16-4ff0-a73d-285bc610009b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145612025-04-30T06:30:26.196ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
145622025-04-30T06:30:26.197ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c7f6759-fac2-41b2-9f57-ba4b8f73a2a2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145632025-04-30T06:30:26.198ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
145642025-04-30T06:30:26.199ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24d7d8f6-7a9e-4daf-bce7-07178f398dac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145652025-04-30T06:30:26.199ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
145662025-04-30T06:30:26.200ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "479a7d2a-db32-487b-a22f-a5f51dd46268", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145672025-04-30T06:30:26.200ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
145682025-04-30T06:30:26.201ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6279fe6b-b912-4f57-91bc-907811616ec3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145692025-04-30T06:30:26.201ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
145702025-04-30T06:30:26.201ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145712025-04-30T06:30:26.201ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145722025-04-30T06:30:26.201ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145732025-04-30T06:30:26.202ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145742025-04-30T06:30:26.205ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145752025-04-30T06:30:26.212ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
145762025-04-30T06:30:26.212ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
145772025-04-30T06:30:26.213ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145782025-04-30T06:30:26.213ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145792025-04-30T06:30:26.213ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145802025-04-30T06:30:26.213ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145812025-04-30T06:30:26.216ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145822025-04-30T06:30:26.223ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
145832025-04-30T06:30:26.223ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145842025-04-30T06:30:26.223ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145852025-04-30T06:30:26.224ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145862025-04-30T06:30:26.282ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
145872025-04-30T06:30:26.282ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
145882025-04-30T06:30:26.283ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
145892025-04-30T06:30:26.283ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145902025-04-30T06:30:26.283ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145912025-04-30T06:30:26.283ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
145922025-04-30T06:30:26.283ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145932025-04-30T06:30:26.284ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:54365 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
145942025-04-30T06:30:26.284ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:54365 req_id = d51580cb-ecf7-4b06-b376-8e13741e0d55 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
145952025-04-30T06:30:26.285ZINFOtest_image_promotion (dropshot): request completed latency_us = 452 local_addr = [::1]:60070 method = GET remote_addr = [::1]:54365 req_id = d51580cb-ecf7-4b06-b376-8e13741e0d55 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
145962025-04-30T06:30:26.285ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d51580cb-ecf7-4b06-b376-8e13741e0d55", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
145972025-04-30T06:30:26.287ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
145982025-04-30T06:30:26.287ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
145992025-04-30T06:30:26.287ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
146002025-04-30T06:30:26.289ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
146012025-04-30T06:30:26.289ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
146022025-04-30T06:30:26.289ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
146032025-04-30T06:30:26.289ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
146042025-04-30T06:30:26.290ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
146052025-04-30T06:30:26.290ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146062025-04-30T06:30:26.295ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
146072025-04-30T06:30:26.295ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146082025-04-30T06:30:26.295ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
146092025-04-30T06:30:26.295ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146102025-04-30T06:30:26.298ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146112025-04-30T06:30:26.303ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
146122025-04-30T06:30:26.303ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
146132025-04-30T06:30:26.303ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
146142025-04-30T06:30:26.304ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
146152025-04-30T06:30:26.307ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146162025-04-30T06:30:26.307ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
146172025-04-30T06:30:26.307ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146182025-04-30T06:30:26.334ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
146192025-04-30T06:30:26.334ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
146202025-04-30T06:30:26.334ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 449.684778ms iteration = 54 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
146212025-04-30T06:30:26.334ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 55 reason = Signaled
146222025-04-30T06:30:26.334ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
146232025-04-30T06:30:26.334ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
146242025-04-30T06:30:26.334ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
146252025-04-30T06:30:26.334ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
146262025-04-30T06:30:26.335ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
146272025-04-30T06:30:26.339ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
146282025-04-30T06:30:26.340ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146292025-04-30T06:30:26.346ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
146302025-04-30T06:30:26.346ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
146312025-04-30T06:30:26.346ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
146322025-04-30T06:30:26.347ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
146332025-04-30T06:30:26.350ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
146342025-04-30T06:30:26.350ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
146352025-04-30T06:30:26.352ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e8 86 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54141 req_id = 88492550-be34-4ed6-bd41-ac6fa1362ee3
146362025-04-30T06:30:26.352ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59526,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54141 req_id = 88492550-be34-4ed6-bd41-ac6fa1362ee3
146372025-04-30T06:30:26.352ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
146382025-04-30T06:30:26.352ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
146392025-04-30T06:30:26.352ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54141 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 88492550-be34-4ed6-bd41-ac6fa1362ee3
146402025-04-30T06:30:26.353ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3491, tv_nsec: 152323544 } })
146412025-04-30T06:30:26.353ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
146422025-04-30T06:30:26.353ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 75 2b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e u+..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59044 req_id = 0b0a5d18-7dbd-4bfc-aa4f-c2776b3b12a2
146432025-04-30T06:30:26.353ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29995,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59044 req_id = 0b0a5d18-7dbd-4bfc-aa4f-c2776b3b12a2
146442025-04-30T06:30:26.354ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
146452025-04-30T06:30:26.354ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59044 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0b0a5d18-7dbd-4bfc-aa4f-c2776b3b12a2
146462025-04-30T06:30:26.416ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
146472025-04-30T06:30:26.417ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
146482025-04-30T06:30:26.417ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
146492025-04-30T06:30:26.417ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146502025-04-30T06:30:26.417ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
146512025-04-30T06:30:26.418ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146522025-04-30T06:30:26.419ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9e d1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57741 req_id = 5ed175c1-0bce-402d-b905-9a344018dbc7
146532025-04-30T06:30:26.419ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40657,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57741 req_id = 5ed175c1-0bce-402d-b905-9a344018dbc7
146542025-04-30T06:30:26.419ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
146552025-04-30T06:30:26.419ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40657, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57741 req_id = 5ed175c1-0bce-402d-b905-9a344018dbc7
146562025-04-30T06:30:26.420ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
146572025-04-30T06:30:26.420ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
146582025-04-30T06:30:26.420ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
146592025-04-30T06:30:26.422ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146602025-04-30T06:30:26.429ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
146612025-04-30T06:30:26.429ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146622025-04-30T06:30:26.429ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
146632025-04-30T06:30:26.430ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146642025-04-30T06:30:26.433ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "63cb6dab-1d4e-44d3-925b-d09ae1e41e70", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146652025-04-30T06:30:26.434ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
146662025-04-30T06:30:26.434ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146672025-04-30T06:30:26.435ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
146682025-04-30T06:30:26.436ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146692025-04-30T06:30:26.438ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "457afba2-a740-4ee5-ae36-851b68e49fe9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146702025-04-30T06:30:26.438ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
146712025-04-30T06:30:26.446ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e974f8c8-cff0-4ed0-bfb4-3a2cda922745", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146722025-04-30T06:30:26.446ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
146732025-04-30T06:30:26.449ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5abc72c6-d364-4227-b80b-cfa964f8b056", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146742025-04-30T06:30:26.449ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
146752025-04-30T06:30:26.451ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00ec86a0-95e7-4cb8-b5b7-233dd1572f1f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146762025-04-30T06:30:26.451ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
146772025-04-30T06:30:26.467ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d8322c9-268e-4663-b8c1-153a483e0439", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146782025-04-30T06:30:26.467ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
146792025-04-30T06:30:26.482ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8641b1fa-5dee-4786-9c0c-a98442762aec", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146802025-04-30T06:30:26.482ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
146812025-04-30T06:30:26.487ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
146822025-04-30T06:30:26.489ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e44b4871-f555-43f4-8dd6-e153ed45e277", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146832025-04-30T06:30:26.490ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
146842025-04-30T06:30:26.491ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2389b07d-7322-4a6f-b399-b395e106e498", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146852025-04-30T06:30:26.492ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
146862025-04-30T06:30:26.494ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fffa12b2-2300-4d7b-99bb-02390c4df265", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146872025-04-30T06:30:26.494ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
146882025-04-30T06:30:26.497ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "683d4eac-f40a-4681-b6eb-8fa941f0742b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146892025-04-30T06:30:26.497ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
146902025-04-30T06:30:26.498ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45d16c39-44a4-48ff-8d9a-e7f9d56d1af8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146912025-04-30T06:30:26.498ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
146922025-04-30T06:30:26.501ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e71d30ac-149a-4c28-93db-adca62272297", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146932025-04-30T06:30:26.501ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
146942025-04-30T06:30:26.503ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "217cd28a-1233-44e2-933a-ec21bae29e9b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146952025-04-30T06:30:26.503ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
146962025-04-30T06:30:26.505ZINFOtest_image_promotion (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
146972025-04-30T06:30:26.505ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e22a444-d241-4125-8e6e-29a1d97b4d47", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
146982025-04-30T06:30:26.505ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
146992025-04-30T06:30:26.505ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
147002025-04-30T06:30:26.506ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 8f1327e9-a116-47f7-bcb5-3d170d3ccafc uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
147012025-04-30T06:30:26.506ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 8f1327e9-a116-47f7-bcb5-3d170d3ccafc resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
147022025-04-30T06:30:26.506ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 8f1327e9-a116-47f7-bcb5-3d170d3ccafc roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
147032025-04-30T06:30:26.507ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 8f1327e9-a116-47f7-bcb5-3d170d3ccafc resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
147042025-04-30T06:30:26.509ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ebee099-a4f4-4eed-ad8c-189473ebaf3b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147052025-04-30T06:30:26.509ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
147062025-04-30T06:30:26.509ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 3562 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 8f1327e9-a116-47f7-bcb5-3d170d3ccafc response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
147072025-04-30T06:30:26.510ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f1327e9-a116-47f7-bcb5-3d170d3ccafc", "content-length": "405", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147082025-04-30T06:30:26.510ZDEBGtest_image_promotion (oximeter): client request body = None method = GET uri = http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
147092025-04-30T06:30:26.511ZTRACtest_image_promotion (dropshot_internal): incoming request local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 4cfd793c-ad8a-4a00-bbc5-0a2051323107 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
147102025-04-30T06:30:26.511ZTRACtest_image_promotion (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 4cfd793c-ad8a-4a00-bbc5-0a2051323107 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
147112025-04-30T06:30:26.511ZDEBGtest_image_promotion (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 4cfd793c-ad8a-4a00-bbc5-0a2051323107 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
147122025-04-30T06:30:26.512ZDEBGtest_image_promotion (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 4cfd793c-ad8a-4a00-bbc5-0a2051323107 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
147132025-04-30T06:30:26.513ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f11ddac-4469-441e-8802-feaebcdfc5c3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147142025-04-30T06:30:26.514ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
147152025-04-30T06:30:26.514ZINFOtest_image_promotion (dropshot_internal): request completed latency_us = 3530 local_addr = 127.0.0.1:45630 method = GET remote_addr = 127.0.0.1:34622 req_id = 4cfd793c-ad8a-4a00-bbc5-0a2051323107 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D
147162025-04-30T06:30:26.514ZDEBGtest_image_promotion (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:45630/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWI1MWFiMC05ZGNlLTQ0ZGEtOTc2NC04ZmQzN2YzYjA0OTgifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cfd793c-ad8a-4a00-bbc5-0a2051323107", "content-length": "29", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147172025-04-30T06:30:26.514ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
147182025-04-30T06:30:26.514ZTRACtest_image_promotion (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = d9b51ab0-9dce-44da-9764-8fd37f3b0498
147192025-04-30T06:30:26.515ZINFOtest_image_promotion (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
147202025-04-30T06:30:26.517ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5d5b454-d98a-4108-b07d-65b6c6573dcb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147212025-04-30T06:30:26.517ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
147222025-04-30T06:30:26.519ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "152b7480-6135-4d52-b5b6-a2540aff6f9d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147232025-04-30T06:30:26.519ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
147242025-04-30T06:30:26.523ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b016a373-4807-41c0-8455-eb5b68abc11b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147252025-04-30T06:30:26.526ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
147262025-04-30T06:30:26.527ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "afaacb8f-df78-441d-a1c2-b0b2d1c88c18", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147272025-04-30T06:30:26.527ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
147282025-04-30T06:30:26.528ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b66218c7-8679-40f5-be43-80d9075afd04", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147292025-04-30T06:30:26.528ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
147302025-04-30T06:30:26.529ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27b88240-7e59-459f-ba66-56919af128cd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147312025-04-30T06:30:26.529ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
147322025-04-30T06:30:26.532ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86acc521-1929-48a0-b734-3419f2d8d800", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147332025-04-30T06:30:26.533ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
147342025-04-30T06:30:26.535ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1689b1da-7470-4bad-9bb2-5d729e32333e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147352025-04-30T06:30:26.535ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
147362025-04-30T06:30:26.537ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32e5b84e-47cb-4437-880b-5f3ceb0ef4b7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147372025-04-30T06:30:26.538ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
147382025-04-30T06:30:26.538ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
147392025-04-30T06:30:26.538ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147402025-04-30T06:30:26.538ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147412025-04-30T06:30:26.538ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
147422025-04-30T06:30:26.539ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147432025-04-30T06:30:26.539ZDEBGtest_image_promotion (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
147442025-04-30T06:30:26.539ZDEBGtest_image_promotion (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
147452025-04-30T06:30:26.545ZTRACtest_image_promotion (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
147462025-04-30T06:30:26.547ZDEBGtest_image_promotion (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
147472025-04-30T06:30:26.547ZTRACtest_image_promotion (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
147482025-04-30T06:30:26.547ZTRACtest_image_promotion (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
147492025-04-30T06:30:26.548ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
147502025-04-30T06:30:26.553ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
147512025-04-30T06:30:26.553ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147522025-04-30T06:30:26.553ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
147532025-04-30T06:30:26.554ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147542025-04-30T06:30:26.557ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147552025-04-30T06:30:26.557ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
147562025-04-30T06:30:26.559ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147572025-04-30T06:30:26.560ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d616a84b-4c5d-42fd-81b1-f794d8343fa9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147582025-04-30T06:30:26.560ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
147592025-04-30T06:30:26.563ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf2361e3-dc52-45ed-bc7a-79b36aa01251", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147602025-04-30T06:30:26.563ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
147612025-04-30T06:30:26.566ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74aa11f9-76a0-4b3a-b453-0a30c7ce54b0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147622025-04-30T06:30:26.566ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
147632025-04-30T06:30:26.570ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eb58944-a8c9-4ca1-9945-3e28f5a933b9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147642025-04-30T06:30:26.570ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
147652025-04-30T06:30:26.571ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6cef12a-2137-4766-8c0a-79b5e5ebdcc3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147662025-04-30T06:30:26.571ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
147672025-04-30T06:30:26.572ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "275577ff-7bb0-4643-b00f-3951c906a0cd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147682025-04-30T06:30:26.573ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
147692025-04-30T06:30:26.574ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2adbf37a-3bdb-4923-831e-26412a7d94a4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147702025-04-30T06:30:26.574ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
147712025-04-30T06:30:26.576ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fb57e06-3547-4d6d-8d22-6fe0b20b9693", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147722025-04-30T06:30:26.576ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
147732025-04-30T06:30:26.577ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6d49e2f-d84c-46ac-92f7-b75f59148b12", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147742025-04-30T06:30:26.578ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
147752025-04-30T06:30:26.580ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bf1634e-b6f2-4b2f-ba6d-bcbf07189c95", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147762025-04-30T06:30:26.580ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
147772025-04-30T06:30:26.583ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "782f636f-9f47-4466-94a8-57b10859bbe8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147782025-04-30T06:30:26.583ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
147792025-04-30T06:30:26.587ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a2f7c7e-99e0-4f65-9077-ae5d0623e3f6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147802025-04-30T06:30:26.587ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
147812025-04-30T06:30:26.591ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7555ae13-dd8e-47da-9289-a2924c924ee9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147822025-04-30T06:30:26.592ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
147832025-04-30T06:30:26.596ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
147842025-04-30T06:30:26.597ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29025a11-fd17-4d5c-9bb9-5be18bf7858d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147852025-04-30T06:30:26.597ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
147862025-04-30T06:30:26.604ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f87b1c8f-2a77-42a1-99c7-3330a674e97b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147872025-04-30T06:30:26.604ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
147882025-04-30T06:30:26.607ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19876509-8f8c-47b1-9d78-be0bd3133d4c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147892025-04-30T06:30:26.607ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
147902025-04-30T06:30:26.608ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "716cfe3f-6a2c-411f-acbb-a8b9526123b5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147912025-04-30T06:30:26.608ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
147922025-04-30T06:30:26.609ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f9ca569-71e6-419b-87bf-37df2549aaa9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147932025-04-30T06:30:26.610ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
147942025-04-30T06:30:26.611ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "693a2627-f871-4cf4-aad4-eb7312ba2517", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
147952025-04-30T06:30:26.611ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
147962025-04-30T06:30:26.612ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147972025-04-30T06:30:26.612ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
147982025-04-30T06:30:26.612ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
147992025-04-30T06:30:26.612ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148002025-04-30T06:30:26.615ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148012025-04-30T06:30:26.621ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
148022025-04-30T06:30:26.621ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
148032025-04-30T06:30:26.621ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148042025-04-30T06:30:26.621ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148052025-04-30T06:30:26.621ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148062025-04-30T06:30:26.622ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148072025-04-30T06:30:26.624ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148082025-04-30T06:30:26.628ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
148092025-04-30T06:30:26.628ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148102025-04-30T06:30:26.628ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148112025-04-30T06:30:26.628ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148122025-04-30T06:30:26.697ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
148132025-04-30T06:30:26.697ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory
148142025-04-30T06:30:26.697ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
148152025-04-30T06:30:26.697ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148162025-04-30T06:30:26.697ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148172025-04-30T06:30:26.698ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
148182025-04-30T06:30:26.699ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148192025-04-30T06:30:26.699ZINFOtest_image_promotion (dropshot): accepted connection local_addr = [::1]:60070 remote_addr = [::1]:51039 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
148202025-04-30T06:30:26.700ZTRACtest_image_promotion (dropshot): incoming request local_addr = [::1]:60070 method = GET remote_addr = [::1]:51039 req_id = 74600aa8-ee5b-44a0-addd-283d5790fe72 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
148212025-04-30T06:30:26.701ZINFOtest_image_promotion (dropshot): request completed latency_us = 898 local_addr = [::1]:60070 method = GET remote_addr = [::1]:51039 req_id = 74600aa8-ee5b-44a0-addd-283d5790fe72 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
148222025-04-30T06:30:26.703ZDEBGtest_image_promotion (BackgroundTasks): client response SledAgent = http://[::1]:60070 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:60070/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "74600aa8-ee5b-44a0-addd-283d5790fe72", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
148232025-04-30T06:30:26.705ZDEBGtest_image_promotion (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
148242025-04-30T06:30:26.705ZDEBGtest_image_promotion (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
148252025-04-30T06:30:26.706ZDEBGtest_image_promotion (BackgroundTasks): finished collection background_task = inventory_collection
148262025-04-30T06:30:26.710ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
148272025-04-30T06:30:26.710ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148282025-04-30T06:30:26.710ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148292025-04-30T06:30:26.710ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148302025-04-30T06:30:26.711ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148312025-04-30T06:30:26.711ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148322025-04-30T06:30:26.723ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
148332025-04-30T06:30:26.723ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148342025-04-30T06:30:26.723ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
148352025-04-30T06:30:26.724ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148362025-04-30T06:30:26.727ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148372025-04-30T06:30:26.727ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
148382025-04-30T06:30:26.728ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148392025-04-30T06:30:26.728ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148402025-04-30T06:30:26.733ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
148412025-04-30T06:30:26.733ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148422025-04-30T06:30:26.734ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148432025-04-30T06:30:26.734ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148442025-04-30T06:30:26.750ZDEBGtest_image_promotion (results-sink): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
148452025-04-30T06:30:26.750ZDEBGtest_image_promotion (results-sink-cluster): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
148462025-04-30T06:30:26.751ZDEBGtest_image_promotion (clickhouse-client): unrolling 2 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880
148472025-04-30T06:30:26.753ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
148482025-04-30T06:30:26.760ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = fields_string
148492025-04-30T06:30:26.760ZTRACtest_image_promotion (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
148502025-04-30T06:30:26.769ZDEBGtest_image_promotion (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a716977f-f822-467e-b942-62720757c880 n_rows = 2 table_name = measurements_i64
148512025-04-30T06:30:26.769ZTRACtest_image_promotion (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
148522025-04-30T06:30:26.770ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
148532025-04-30T06:30:26.771ZWARNtest_image_promotion (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
148542025-04-30T06:30:26.771ZWARNtest_image_promotion (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
148552025-04-30T06:30:26.771ZDEBGtest_image_promotion (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 437.314221ms iteration = 55 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
148562025-04-30T06:30:26.771ZDEBGtest_image_promotion (BackgroundTasks): activating background_task = inventory_collection iteration = 56 reason = Signaled
148572025-04-30T06:30:26.771ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
148582025-04-30T06:30:26.771ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148592025-04-30T06:30:26.771ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148602025-04-30T06:30:26.771ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148612025-04-30T06:30:26.772ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148622025-04-30T06:30:26.774ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148632025-04-30T06:30:26.780ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
148642025-04-30T06:30:26.780ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148652025-04-30T06:30:26.780ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148662025-04-30T06:30:26.780ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148672025-04-30T06:30:26.783ZDEBGtest_image_promotion (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(a1711333-5886-4224-9910-e4f59cbe995f, 0)]
148682025-04-30T06:30:26.783ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
148692025-04-30T06:30:26.784ZTRACtest_image_promotion (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ff 0c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49302 req_id = 6436ac8e-46ce-40cf-bdc4-483969a9bea6
148702025-04-30T06:30:26.784ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65292,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49302 req_id = 6436ac8e-46ce-40cf-bdc4-483969a9bea6
148712025-04-30T06:30:26.785ZDEBGtest_image_promotion (store): query key key = _mgs._tcp
148722025-04-30T06:30:26.785ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
148732025-04-30T06:30:26.785ZDEBGtest_image_promotion (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49302 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6436ac8e-46ce-40cf-bdc4-483969a9bea6
148742025-04-30T06:30:26.786ZDEBGtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59233, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3491, tv_nsec: 585196705 } })
148752025-04-30T06:30:26.788ZTRACtest_image_promotion (DnsResolver): lookup_service_targets: looking up SRV target
148762025-04-30T06:30:26.789ZTRACtest_image_promotion (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ee b7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60269 req_id = 88461e87-be21-4241-88e6-451ad07c2be0
148772025-04-30T06:30:26.789ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61111,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60269 req_id = 88461e87-be21-4241-88e6-451ad07c2be0
148782025-04-30T06:30:26.790ZDEBGtest_image_promotion (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
148792025-04-30T06:30:26.790ZDEBGtest_image_promotion (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60269 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 88461e87-be21-4241-88e6-451ad07c2be0
148802025-04-30T06:30:26.852ZTRACtest_image_promotion (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
148812025-04-30T06:30:26.853ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
148822025-04-30T06:30:26.853ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148832025-04-30T06:30:26.853ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148842025-04-30T06:30:26.853ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
148852025-04-30T06:30:26.855ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148862025-04-30T06:30:26.856ZWARNtest_image_promotion (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
148872025-04-30T06:30:26.856ZTRACtest_image_promotion (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: cb 54 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .T..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53025 req_id = 4518c1d5-542c-4453-b96f-563a11a66ef0
148882025-04-30T06:30:26.856ZDEBGtest_image_promotion (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52052,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53025 req_id = 4518c1d5-542c-4453-b96f-563a11a66ef0
148892025-04-30T06:30:26.857ZDEBGtest_image_promotion (store): query key key = _clickhouse-admin-keeper._tcp
148902025-04-30T06:30:26.857ZERROtest_image_promotion (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52052, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53025 req_id = 4518c1d5-542c-4453-b96f-563a11a66ef0
148912025-04-30T06:30:26.858ZDEBGtest_image_promotion (BackgroundTasks): begin collection background_task = inventory_collection
148922025-04-30T06:30:26.858ZDEBGtest_image_promotion (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:59233
148932025-04-30T06:30:26.858ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/ignition
148942025-04-30T06:30:26.859ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148952025-04-30T06:30:26.866ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
148962025-04-30T06:30:26.867ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148972025-04-30T06:30:26.867ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
148982025-04-30T06:30:26.867ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148992025-04-30T06:30:26.870ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d78f807b-fac5-4f2e-af65-9f14a25fe395", "content-length": "835", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149002025-04-30T06:30:26.870ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0
149012025-04-30T06:30:26.871ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
149022025-04-30T06:30:26.871ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
149032025-04-30T06:30:26.871ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
149042025-04-30T06:30:26.874ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffc11d29-aa8e-47b5-84e0-2543ef317153", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149052025-04-30T06:30:26.874ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0
149062025-04-30T06:30:26.877ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "47711bd7-f651-48c7-9d22-2af2dce4944e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149072025-04-30T06:30:26.877ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1
149082025-04-30T06:30:26.879ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ed37891-014b-4758-be64-fee7246f4da6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149092025-04-30T06:30:26.879ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0
149102025-04-30T06:30:26.882ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "daf66497-7bdb-4e2a-b067-2b644197171f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149112025-04-30T06:30:26.882ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1
149122025-04-30T06:30:26.884ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d42186c-17f0-4872-ba7a-ab7b07ab4293", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149132025-04-30T06:30:26.884ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0
149142025-04-30T06:30:26.888ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0a6e379-ce2b-4f72-adc0-d5e732f14615", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149152025-04-30T06:30:26.888ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1
149162025-04-30T06:30:26.891ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "827b418e-0490-422c-a81c-7e1314b9b3ef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149172025-04-30T06:30:26.892ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cmpa
149182025-04-30T06:30:26.893ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b78babef-61f8-4132-8fdc-e61a915332fd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149192025-04-30T06:30:26.893ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
149202025-04-30T06:30:26.894ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ea7cd55-9831-4a02-99ef-8009e531deb0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149212025-04-30T06:30:26.894ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
149222025-04-30T06:30:26.895ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "47ac99e5-3fbe-4c43-9ea9-86c42a94030a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149232025-04-30T06:30:26.895ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/0/component/rot/cfpa
149242025-04-30T06:30:26.897ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4846df7b-1940-432f-9c71-ce93710eeb3b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149252025-04-30T06:30:26.897ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1
149262025-04-30T06:30:26.899ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed39452e-8e3f-40a1-8c50-c11da5af0589", "content-length": "734", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149272025-04-30T06:30:26.900ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0
149282025-04-30T06:30:26.905ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c91840d0-c567-4216-9acc-4a80e582aa1d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149292025-04-30T06:30:26.905ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1
149302025-04-30T06:30:26.906ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
149312025-04-30T06:30:26.908ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9b5ec50-f52a-4e27-bb80-3365c0c731f7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149322025-04-30T06:30:26.909ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0
149332025-04-30T06:30:26.910ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "245a2014-25c6-4deb-bf1a-20a9d5b3e800", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149342025-04-30T06:30:26.911ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1
149352025-04-30T06:30:26.913ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c1410ad-4510-419d-8fa7-fcc9101d5b35", "content-length": "170", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149362025-04-30T06:30:26.913ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0
149372025-04-30T06:30:26.917ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be9a45cf-77e7-40a2-9585-7a01cbc2b399", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149382025-04-30T06:30:26.917ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1
149392025-04-30T06:30:26.920ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7ae1f4d-6b24-4a51-9b1a-c3d7cb00eeec", "content-length": "179", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149402025-04-30T06:30:26.920ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cmpa
149412025-04-30T06:30:26.921ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05580dd0-bd98-4cf1-a3aa-e839cd7a5e4c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149422025-04-30T06:30:26.922ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
149432025-04-30T06:30:26.923ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf6d8f17-276c-445b-bc6c-563935520107", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149442025-04-30T06:30:26.923ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
149452025-04-30T06:30:26.923ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d5f928e-d586-48a2-9763-ab6ac46e316f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149462025-04-30T06:30:26.924ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/switch/1/component/rot/cfpa
149472025-04-30T06:30:26.925ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c81c0d5-1cd8-420d-bc6d-cb0e7786582c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149482025-04-30T06:30:26.925ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0
149492025-04-30T06:30:26.926ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f695dda-3902-4978-898d-bfbbacf592e8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149502025-04-30T06:30:26.926ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0
149512025-04-30T06:30:26.928ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4b44865-08a4-4be6-9435-c72c1f82a333", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149522025-04-30T06:30:26.928ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1
149532025-04-30T06:30:26.930ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1a248b3-1d9f-4adc-8816-c6bc7a1509d1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149542025-04-30T06:30:26.930ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0
149552025-04-30T06:30:26.932ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dafff9b-e706-4122-8842-0033b0c8ec06", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149562025-04-30T06:30:26.932ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1
149572025-04-30T06:30:26.934ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59faafe6-2446-4d5d-b5c8-cb7b8f0ee968", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149582025-04-30T06:30:26.934ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0
149592025-04-30T06:30:26.936ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "921618c0-8c07-4ffe-9a83-77410df4cad2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149602025-04-30T06:30:26.936ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1
149612025-04-30T06:30:26.938ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "178611bb-3fd2-45ab-b3f1-b649f9f8a6ec", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149622025-04-30T06:30:26.938ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cmpa
149632025-04-30T06:30:26.939ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfdc2fe4-833c-4a27-9235-697a44a2baa6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149642025-04-30T06:30:26.939ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
149652025-04-30T06:30:26.940ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18cac63e-4340-487b-9094-0fc2df6c90be", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149662025-04-30T06:30:26.940ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
149672025-04-30T06:30:26.941ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8da01b05-197c-455e-a971-2215893304dd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149682025-04-30T06:30:26.942ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/0/component/rot/cfpa
149692025-04-30T06:30:26.943ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "102d6796-5523-444d-a7e6-688c947a28e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149702025-04-30T06:30:26.943ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1
149712025-04-30T06:30:26.944ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35940a87-a0f1-4bee-a996-ce7f1e494876", "content-length": "723", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149722025-04-30T06:30:26.944ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0
149732025-04-30T06:30:26.946ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94391722-9b99-455e-b19f-55a021128900", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149742025-04-30T06:30:26.946ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1
149752025-04-30T06:30:26.948ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f14a9feb-46e5-4427-8dc6-ec27ccc4ad61", "content-length": "109", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149762025-04-30T06:30:26.948ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0
149772025-04-30T06:30:26.957ZTRACtest_image_promotion (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
149782025-04-30T06:30:26.957ZTRACtest_image_promotion (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
149792025-04-30T06:30:26.957ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
149802025-04-30T06:30:26.957ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
149812025-04-30T06:30:26.958ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
149822025-04-30T06:30:26.963ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
149832025-04-30T06:30:26.970ZDEBGtest_image_promotion (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
149842025-04-30T06:30:26.970ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
149852025-04-30T06:30:26.970ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
149862025-04-30T06:30:26.971ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
149872025-04-30T06:30:26.978ZTRACtest_image_promotion (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
149882025-04-30T06:30:26.978ZDEBGtest_image_promotion (InternalApi): roles roles = RoleSet { roles: {} }
149892025-04-30T06:30:26.979ZDEBGtest_image_promotion (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
149902025-04-30T06:30:26.991ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fdce97e-1ed6-4e85-9052-09ee72a4d239", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:25 GMT"} })
149912025-04-30T06:30:26.991ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1
149922025-04-30T06:30:27.004ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c82afef8-b88a-4f7e-bd6f-f6f65064163e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:30:27 GMT"} })
149932025-04-30T06:30:27.004ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0
149942025-04-30T06:30:27.007ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "24efc6b3-1f96-4a7f-93cf-fda213bfe23c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:27 GMT"} })
149952025-04-30T06:30:27.007ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1
149962025-04-30T06:30:27.009ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aff74141-62a8-455a-88bb-f251c64c9c6b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:30:27 GMT"} })
149972025-04-30T06:30:27.010ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cmpa
149982025-04-30T06:30:27.011ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6a6dad7-2c3a-438f-a6ee-5f3926a48644", "content-length": "702", "date": "Wed, 30 Apr 2025 06:30:27 GMT"} })
149992025-04-30T06:30:27.011ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
150002025-04-30T06:30:27.013ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac6133be-fc91-4248-a8c5-656d5308f745", "content-length": "727", "date": "Wed, 30 Apr 2025 06:30:27 GMT"} })
150012025-04-30T06:30:27.013ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
150022025-04-30T06:30:27.016ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3beb37d-6772-4511-a998-9d653c9fc440", "content-length": "729", "date": "Wed, 30 Apr 2025 06:30:27 GMT"} })
150032025-04-30T06:30:27.016ZDEBGtest_image_promotion (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:59233 method = GET uri = http://[::1]:59233/sp/sled/1/component/rot/cfpa
150042025-04-30T06:30:27.017ZDEBGtest_image_promotion: attempt to fetch latest inventory collection result = Ok("found")
150052025-04-30T06:30:27.018ZDEBGtest_image_promotion (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:59233 result = Ok(Response { url: "http://[::1]:59233/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "380b6de4-f351-46ed-8d5e-753414771f73", "content-length": "728", "date": "Wed, 30 Apr 2025 06:30:27 GMT"} })
150062025-04-30T06:30:27.018ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
150072025-04-30T06:30:27.018ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
150082025-04-30T06:30:27.018ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
150092025-04-30T06:30:27.018ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
150102025-04-30T06:30:27.021ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
150112025-04-30T06:30:27.023ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
150122025-04-30T06:30:27.031ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
150132025-04-30T06:30:27.032ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
150142025-04-30T06:30:27.032ZTRACtest_image_promotion (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
150152025-04-30T06:30:27.032ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
150162025-04-30T06:30:27.032ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
150172025-04-30T06:30:27.032ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
150182025-04-30T06:30:27.035ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
150192025-04-30T06:30:27.039ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
150202025-04-30T06:30:27.039ZTRACtest_image_promotion (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
150212025-04-30T06:30:27.039ZDEBGtest_image_promotion (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
150222025-04-30T06:30:27.040ZDEBGtest_image_promotion (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
150232025-04-30T06:30:27.093ZDEBGtest_image_promotion (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:60070
150242025-04-30T06:30:27.093ZDEBGtest_image_promotion (BackgroundTasks): client request SledAgent = http://[::1]:60070 background_task = inventory_collection body = None method = GET uri = http://[::1]:60070/inventory