Max level shown:
1 [ Mar 21 02:04:12 Enabled. ]
2 [ Mar 21 02:04:12 Rereading configuration. ]
3 [ Mar 21 02:04:14 Rereading configuration. ]
4 [ Mar 21 02:04:23 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/omicron-nexus/bin/nexus /var/svc/manifest/site/nexus/config.toml &"). ]
5 [ Mar 21 02:04:23 Method "start" exited with status 0. ]
6 note: configured to log to "/dev/stdout"
72025-03-21T02:04:23.550ZDEBGnexus: registered DTrace probes
82025-03-21T02:04:23.552ZINFOnexus: setting up nexus server file = nexus/src/lib.rs:78
92025-03-21T02:04:23.563ZINFOnexus (ServerContext): registering Oso class class = Action file = nexus/auth/src/authz/oso_generic.rs:68
102025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = AnyActor file = nexus/auth/src/authz/oso_generic.rs:68
112025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = AuthenticatedActor file = nexus/auth/src/authz/oso_generic.rs:68
122025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = BlueprintConfig file = nexus/auth/src/authz/oso_generic.rs:68
132025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = Database file = nexus/auth/src/authz/oso_generic.rs:68
142025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = DnsConfig file = nexus/auth/src/authz/oso_generic.rs:68
152025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = Fleet file = nexus/auth/src/authz/oso_generic.rs:68
162025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = Inventory file = nexus/auth/src/authz/oso_generic.rs:68
172025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = IpPoolList file = nexus/auth/src/authz/oso_generic.rs:68
182025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = ConsoleSessionList file = nexus/auth/src/authz/oso_generic.rs:68
192025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = DeviceAuthRequestList file = nexus/auth/src/authz/oso_generic.rs:68
202025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = SiloCertificateList file = nexus/auth/src/authz/oso_generic.rs:68
212025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = SiloIdentityProviderList file = nexus/auth/src/authz/oso_generic.rs:68
222025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = SiloUserList file = nexus/auth/src/authz/oso_generic.rs:68
232025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = TargetReleaseConfig file = nexus/auth/src/authz/oso_generic.rs:68
242025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = Project file = nexus/auth/src/authz/oso_generic.rs:68
252025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = Disk file = nexus/auth/src/authz/oso_generic.rs:68
262025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = Snapshot file = nexus/auth/src/authz/oso_generic.rs:68
272025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = ProjectImage file = nexus/auth/src/authz/oso_generic.rs:68
282025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = AffinityGroup file = nexus/auth/src/authz/oso_generic.rs:68
292025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = AntiAffinityGroup file = nexus/auth/src/authz/oso_generic.rs:68
302025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = Instance file = nexus/auth/src/authz/oso_generic.rs:68
312025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = IpPool file = nexus/auth/src/authz/oso_generic.rs:68
322025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = InstanceNetworkInterface file = nexus/auth/src/authz/oso_generic.rs:68
332025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = Vpc file = nexus/auth/src/authz/oso_generic.rs:68
342025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = VpcRouter file = nexus/auth/src/authz/oso_generic.rs:68
352025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = InternetGateway file = nexus/auth/src/authz/oso_generic.rs:68
362025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = InternetGatewayIpPool file = nexus/auth/src/authz/oso_generic.rs:68
372025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = InternetGatewayIpAddress file = nexus/auth/src/authz/oso_generic.rs:68
382025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = RouterRoute file = nexus/auth/src/authz/oso_generic.rs:68
392025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = VpcSubnet file = nexus/auth/src/authz/oso_generic.rs:68
402025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = FloatingIp file = nexus/auth/src/authz/oso_generic.rs:68
412025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = Image file = nexus/auth/src/authz/oso_generic.rs:68
422025-03-21T02:04:23.564ZINFOnexus (ServerContext): registering Oso class class = SiloImage file = nexus/auth/src/authz/oso_generic.rs:68
432025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = AddressLot file = nexus/auth/src/authz/oso_generic.rs:68
442025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = Blueprint file = nexus/auth/src/authz/oso_generic.rs:68
452025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = LoopbackAddress file = nexus/auth/src/authz/oso_generic.rs:68
462025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = Certificate file = nexus/auth/src/authz/oso_generic.rs:68
472025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = ConsoleSession file = nexus/auth/src/authz/oso_generic.rs:68
482025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = DeviceAuthRequest file = nexus/auth/src/authz/oso_generic.rs:68
492025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = DeviceAccessToken file = nexus/auth/src/authz/oso_generic.rs:68
502025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = PhysicalDisk file = nexus/auth/src/authz/oso_generic.rs:68
512025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = Rack file = nexus/auth/src/authz/oso_generic.rs:68
522025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = RoleBuiltin file = nexus/auth/src/authz/oso_generic.rs:68
532025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = SshKey file = nexus/auth/src/authz/oso_generic.rs:68
542025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = Silo file = nexus/auth/src/authz/oso_generic.rs:68
552025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = SiloUser file = nexus/auth/src/authz/oso_generic.rs:68
562025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = SiloGroup file = nexus/auth/src/authz/oso_generic.rs:68
572025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = SupportBundle file = nexus/auth/src/authz/oso_generic.rs:68
582025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = IdentityProvider file = nexus/auth/src/authz/oso_generic.rs:68
592025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = SamlIdentityProvider file = nexus/auth/src/authz/oso_generic.rs:68
602025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = Sled file = nexus/auth/src/authz/oso_generic.rs:68
612025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = TufRepo file = nexus/auth/src/authz/oso_generic.rs:68
622025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = TufArtifact file = nexus/auth/src/authz/oso_generic.rs:68
632025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = Zpool file = nexus/auth/src/authz/oso_generic.rs:68
642025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = Service file = nexus/auth/src/authz/oso_generic.rs:68
652025-03-21T02:04:23.565ZINFOnexus (ServerContext): registering Oso class class = UserBuiltin file = nexus/auth/src/authz/oso_generic.rs:68
662025-03-21T02:04:23.565ZINFOnexus (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n file = nexus/auth/src/authz/oso_generic.rs:89
672025-03-21T02:04:23.588ZINFOnexus (ServerContext): Setting up resolver using DNS servers for subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344::, width: 48 } } file = nexus/src/context.rs:222
682025-03-21T02:04:23.591ZINFOnexus (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
692025-03-21T02:04:23.596ZINFOnexus (ServerContext): Setting up qorb database pool from DNS dns_addrs = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = nexus/src/context.rs:278
702025-03-21T02:04:23.604ZDEBGnexus (ServerContext): registered USDT probes
712025-03-21T02:04:23.627ZINFOnexus: Database schema version is up to date desired_version = 130.0.0 file = nexus/db-queries/src/db/datastore/db_metadata.rs:144 found_version = 130.0.0
722025-03-21T02:04:23.630ZINFOnexus (SEC): SEC running file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/steno-0.4.1/src/sec.rs:813 sec_id = e05eb799-8cf0-4702-bd3b-9994ae86118a
732025-03-21T02:04:23.632ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 972, tv_nsec: 259875307 } })
742025-03-21T02:04:23.634ZINFOnexus (Nexus): Determining switch slots managed by switch zones file = nexus/src/app/mod.rs:1171
752025-03-21T02:04:23.692ZINFOnexus (Nexus): determining switch slot managed by dendrite zone file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
762025-03-21T02:04:23.692ZDEBGnexus (MgsClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
772025-03-21T02:04:23.696ZDEBGnexus (MgsClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6066970-0f5e-42d1-a9c4-17f5103964c3", "content-length": "26", "date": "Fri, 21 Mar 2025 02:04:23 GMT"} })
782025-03-21T02:04:23.697ZINFOnexus (Nexus): identified switch slot for dendrite zone file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
792025-03-21T02:04:23.698ZINFOnexus (Nexus): completed mapping dendrite zones to switch slots file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
802025-03-21T02:04:23.731ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 972, tv_nsec: 358598256 } })
812025-03-21T02:04:23.731ZINFOnexus (Nexus): Determining switch slots managed by switch zones file = nexus/src/app/mod.rs:1171
822025-03-21T02:04:23.773ZINFOnexus (Nexus): determining switch slot managed by dendrite zone file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
832025-03-21T02:04:23.773ZDEBGnexus (MgsClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
842025-03-21T02:04:23.773ZDEBGnexus (MgsClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1106ac73-524f-474e-8599-c2e47b017c32", "content-length": "26", "date": "Fri, 21 Mar 2025 02:04:23 GMT"} })
852025-03-21T02:04:23.774ZINFOnexus (Nexus): identified switch slot for dendrite zone file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
862025-03-21T02:04:23.774ZINFOnexus (Nexus): completed mapping dendrite zones to switch slots file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
872025-03-21T02:04:23.847ZDEBGnexus (clickhouse-client): registered USDT probes id = 2b7daace-0362-4ae6-9c71-ae7f2f82d600
882025-03-21T02:04:23.863ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
892025-03-21T02:04:23.875ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
902025-03-21T02:04:23.878ZDEBGnexus (DataLoader): attempting to create built-in users
912025-03-21T02:04:23.879ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
922025-03-21T02:04:23.879ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
932025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /bgtasks versions = all
942025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /bgtasks/activate versions = all
952025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /bgtasks/view/{bgtask_name} versions = all
962025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /clickhouse/policy versions = all
972025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /clickhouse/policy versions = all
982025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
992025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1002025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1012025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1022025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1032025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /demo-saga versions = all
1042025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1052025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /deployment/blueprints/all versions = all
1062025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1072025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1082025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /deployment/blueprints/import versions = all
1092025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /deployment/blueprints/regenerate versions = all
1102025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /deployment/blueprints/target versions = all
1112025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /deployment/blueprints/target versions = all
1122025-03-21T02:04:23.896ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = PUT path = /deployment/blueprints/target/enabled versions = all
1132025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1142025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = PUT path = /disks/{disk_id} versions = all
1152025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /instances/{instance_id}/migrate versions = all
1162025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /metrics/collectors versions = all
1172025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1182025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /metrics/producers versions = all
1192025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1202025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /physical-disk/expunge versions = all
1212025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /probes/{sled} versions = all
1222025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1232025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /sagas versions = all
1242025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /sagas/{saga_id} versions = all
1252025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /sled-agents/{sled_id} versions = all
1262025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /sled-agents/{sled_id} versions = all
1272025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1282025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /sleds/add versions = all
1292025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /sleds/expunge versions = all
1302025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = GET path = /sleds/uninitialized versions = all
1312025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = PUT path = /switch/{switch_id} versions = all
1322025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = PUT path = /vmms/{propolis_id} versions = all
1332025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::c]:12221 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1342025-03-21T02:04:23.897ZINFOnexus (dropshot_internal): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [fd00:1122:3344:101::c]:12221
1352025-03-21T02:04:23.897ZDEBGnexus (dropshot_internal): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::c]:12221
1362025-03-21T02:04:23.897ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
1372025-03-21T02:04:23.897ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
1382025-03-21T02:04:23.906ZINFOnexus (DataLoader): created 0 built-in users file = nexus/db-queries/src/db/datastore/silo_user.rs:394
1392025-03-21T02:04:23.906ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1402025-03-21T02:04:23.907ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1412025-03-21T02:04:23.908ZDEBGnexus (DataLoader): attempting to create built-in roles
1422025-03-21T02:04:23.909ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1432025-03-21T02:04:23.909ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1442025-03-21T02:04:23.922ZWARNnexus: Cannot look up rack: Object (of type ById(9e10fa12-3248-42fd-baa3-e5de51ba7745)) not found: rack file = nexus/src/app/rack.rs:776
1452025-03-21T02:04:23.935ZINFOnexus (DataLoader): created 0 built-in roles file = nexus/db-queries/src/db/datastore/role.rs:84
1462025-03-21T02:04:23.935ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1472025-03-21T02:04:23.937ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1482025-03-21T02:04:23.937ZDEBGnexus (DataLoader): attempting to create built-in role assignments
1492025-03-21T02:04:23.937ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1502025-03-21T02:04:23.937ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1512025-03-21T02:04:25.409ZINFOnexus (DataLoader): created 0 built-in role assignments file = nexus/db-queries/src/db/datastore/role.rs:113
1522025-03-21T02:04:25.410ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1532025-03-21T02:04:25.411ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1542025-03-21T02:04:25.411ZDEBGnexus (DataLoader): attempting to create built-in silos
1552025-03-21T02:04:25.411ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1562025-03-21T02:04:25.411ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1572025-03-21T02:04:25.474ZINFOnexus (DataLoader): created 0 built-in silos file = nexus/db-queries/src/db/datastore/silo.rs:93
1582025-03-21T02:04:25.474ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1592025-03-21T02:04:25.475ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1602025-03-21T02:04:25.479ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1612025-03-21T02:04:25.480ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1622025-03-21T02:04:25.480ZDEBGnexus (DataLoader): attempting to create built-in projects
1632025-03-21T02:04:25.480ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1642025-03-21T02:04:25.480ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1652025-03-21T02:04:25.506ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1662025-03-21T02:04:25.506ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1672025-03-21T02:04:25.549ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1682025-03-21T02:04:25.549ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1692025-03-21T02:04:25.583ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1702025-03-21T02:04:25.585ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
1712025-03-21T02:04:25.585ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1722025-03-21T02:04:25.585ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1732025-03-21T02:04:25.631ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1742025-03-21T02:04:25.631ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1752025-03-21T02:04:25.661ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1762025-03-21T02:04:25.663ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
1772025-03-21T02:04:25.663ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1782025-03-21T02:04:25.663ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1792025-03-21T02:04:25.698ZINFOnexus (DataLoader): created built-in services project file = nexus/db-queries/src/db/datastore/project.rs:122
1802025-03-21T02:04:25.698ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1812025-03-21T02:04:25.699ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1822025-03-21T02:04:25.699ZDEBGnexus (DataLoader): attempting to create built-in VPCs
1832025-03-21T02:04:25.699ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1842025-03-21T02:04:25.700ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1852025-03-21T02:04:25.703ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1862025-03-21T02:04:25.703ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1872025-03-21T02:04:25.708ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1882025-03-21T02:04:25.708ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1892025-03-21T02:04:25.710ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1902025-03-21T02:04:25.710ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1912025-03-21T02:04:25.721ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1922025-03-21T02:04:25.721ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1932025-03-21T02:04:25.755ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1942025-03-21T02:04:25.757ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
1952025-03-21T02:04:25.757ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1962025-03-21T02:04:25.758ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1972025-03-21T02:04:25.759ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1982025-03-21T02:04:25.759ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1992025-03-21T02:04:25.792ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2002025-03-21T02:04:25.792ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2012025-03-21T02:04:25.793ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2022025-03-21T02:04:25.796ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2032025-03-21T02:04:25.796ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2042025-03-21T02:04:25.796ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2052025-03-21T02:04:25.848ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2062025-03-21T02:04:25.848ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2072025-03-21T02:04:25.849ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2082025-03-21T02:04:25.850ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2092025-03-21T02:04:25.851ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2102025-03-21T02:04:25.851ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2112025-03-21T02:04:25.853ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2122025-03-21T02:04:25.856ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
2132025-03-21T02:04:25.857ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2142025-03-21T02:04:25.857ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2152025-03-21T02:04:25.923ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
2162025-03-21T02:04:25.923ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
2172025-03-21T02:04:25.925ZWARNnexus: Cannot look up rack: Object (of type ById(9e10fa12-3248-42fd-baa3-e5de51ba7745)) not found: rack file = nexus/src/app/rack.rs:776
2182025-03-21T02:04:25.934ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2192025-03-21T02:04:25.934ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2202025-03-21T02:04:25.955ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2212025-03-21T02:04:25.956ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2222025-03-21T02:04:25.957ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2232025-03-21T02:04:25.958ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2242025-03-21T02:04:25.959ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2252025-03-21T02:04:25.959ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2262025-03-21T02:04:25.960ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2272025-03-21T02:04:25.961ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2282025-03-21T02:04:25.962ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2292025-03-21T02:04:25.962ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2302025-03-21T02:04:25.963ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2312025-03-21T02:04:25.963ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2322025-03-21T02:04:25.965ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2332025-03-21T02:04:25.969ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
2342025-03-21T02:04:25.969ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2352025-03-21T02:04:25.969ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2362025-03-21T02:04:25.970ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2372025-03-21T02:04:25.971ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2382025-03-21T02:04:25.972ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2392025-03-21T02:04:25.972ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2402025-03-21T02:04:25.973ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2412025-03-21T02:04:25.977ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
2422025-03-21T02:04:25.977ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2432025-03-21T02:04:25.977ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2442025-03-21T02:04:26.045ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2452025-03-21T02:04:26.046ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2462025-03-21T02:04:26.047ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2472025-03-21T02:04:26.048ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2482025-03-21T02:04:26.049ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2492025-03-21T02:04:26.049ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2502025-03-21T02:04:26.050ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2512025-03-21T02:04:26.054ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
2522025-03-21T02:04:26.054ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2532025-03-21T02:04:26.054ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2542025-03-21T02:04:26.231ZDEBGnexus (DataLoader): attempting to create built-in VPC firewall rules
2552025-03-21T02:04:26.231ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2562025-03-21T02:04:26.231ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-03-21T02:04:26.233ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2582025-03-21T02:04:26.234ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2592025-03-21T02:04:26.235ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2602025-03-21T02:04:26.235ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2612025-03-21T02:04:26.237ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2622025-03-21T02:04:26.237ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2632025-03-21T02:04:26.239ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2642025-03-21T02:04:26.239ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2652025-03-21T02:04:26.240ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2662025-03-21T02:04:26.240ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2672025-03-21T02:04:26.242ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2682025-03-21T02:04:26.245ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
2692025-03-21T02:04:26.245ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2702025-03-21T02:04:26.245ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2712025-03-21T02:04:26.246ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2722025-03-21T02:04:26.246ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2732025-03-21T02:04:26.247ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2742025-03-21T02:04:26.247ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2752025-03-21T02:04:26.249ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2762025-03-21T02:04:26.252ZDEBGnexus (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
2772025-03-21T02:04:26.252ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2782025-03-21T02:04:26.252ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2792025-03-21T02:04:26.317ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2802025-03-21T02:04:26.317ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2812025-03-21T02:04:26.318ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2822025-03-21T02:04:26.319ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2832025-03-21T02:04:26.320ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2842025-03-21T02:04:26.320ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2852025-03-21T02:04:26.321ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2862025-03-21T02:04:26.324ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
2872025-03-21T02:04:26.324ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2882025-03-21T02:04:26.324ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2892025-03-21T02:04:26.424ZINFOnexus (DataLoader): created built-in VPC firewall rules file = nexus/db-queries/src/db/datastore/vpc.rs:284
2902025-03-21T02:04:26.424ZDEBGnexus (DataLoader): attempting to create built-in VPC Subnets
2912025-03-21T02:04:26.424ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2922025-03-21T02:04:26.425ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2932025-03-21T02:04:26.443ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2942025-03-21T02:04:26.443ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2952025-03-21T02:04:26.445ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2962025-03-21T02:04:26.445ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2972025-03-21T02:04:26.448ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2982025-03-21T02:04:26.448ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2992025-03-21T02:04:26.450ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3002025-03-21T02:04:26.450ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3012025-03-21T02:04:26.451ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3022025-03-21T02:04:26.452ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3032025-03-21T02:04:26.453ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3042025-03-21T02:04:26.456ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3052025-03-21T02:04:26.456ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3062025-03-21T02:04:26.456ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3072025-03-21T02:04:26.459ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3082025-03-21T02:04:26.460ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3092025-03-21T02:04:26.461ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3102025-03-21T02:04:26.461ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3112025-03-21T02:04:26.462ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3122025-03-21T02:04:26.462ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3132025-03-21T02:04:26.464ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3142025-03-21T02:04:26.464ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-03-21T02:04:26.465ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3162025-03-21T02:04:26.465ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3172025-03-21T02:04:26.466ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3182025-03-21T02:04:26.467ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3192025-03-21T02:04:26.468ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3202025-03-21T02:04:26.472ZDEBGnexus (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ById(001de000-c470-4000-8000-000000000001) } result = Ok(())
3212025-03-21T02:04:26.472ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3222025-03-21T02:04:26.473ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3232025-03-21T02:04:26.474ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3242025-03-21T02:04:26.474ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3252025-03-21T02:04:26.476ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3262025-03-21T02:04:26.476ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3272025-03-21T02:04:26.480ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3282025-03-21T02:04:26.480ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3292025-03-21T02:04:26.482ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3302025-03-21T02:04:26.482ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3312025-03-21T02:04:26.484ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3322025-03-21T02:04:26.484ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-03-21T02:04:26.485ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3342025-03-21T02:04:26.485ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3352025-03-21T02:04:26.487ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3362025-03-21T02:04:26.490ZDEBGnexus (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ById(001de000-c470-4000-8000-000000000002) } result = Ok(())
3372025-03-21T02:04:26.490ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3382025-03-21T02:04:26.490ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3392025-03-21T02:04:26.492ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3402025-03-21T02:04:26.492ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3412025-03-21T02:04:26.494ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3422025-03-21T02:04:26.494ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3432025-03-21T02:04:26.495ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3442025-03-21T02:04:26.496ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3452025-03-21T02:04:26.497ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3462025-03-21T02:04:26.497ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3472025-03-21T02:04:26.498ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3482025-03-21T02:04:26.498ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3492025-03-21T02:04:26.500ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3502025-03-21T02:04:26.500ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-03-21T02:04:26.501ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3522025-03-21T02:04:26.505ZDEBGnexus (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-c470-4000-8000-000000000003, lookup_type: ById(001de000-c470-4000-8000-000000000003) } result = Ok(())
3532025-03-21T02:04:26.505ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3542025-03-21T02:04:26.505ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3552025-03-21T02:04:26.511ZINFOnexus (DataLoader): created built-in services vpc subnets file = nexus/db-queries/src/db/datastore/vpc.rs:350
3562025-03-21T02:04:26.511ZINFOnexus (DataLoader): created built-in services vpc file = nexus/db-queries/src/db/datastore/vpc.rs:226
3572025-03-21T02:04:26.511ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3582025-03-21T02:04:26.512ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3592025-03-21T02:04:26.512ZDEBGnexus (DataLoader): attempting to create silo users
3602025-03-21T02:04:26.512ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3612025-03-21T02:04:26.513ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3622025-03-21T02:04:26.529ZINFOnexus (DataLoader): created 0 silo users file = nexus/db-queries/src/db/datastore/silo_user.rs:419
3632025-03-21T02:04:26.529ZDEBGnexus (DataLoader): attempting to create silo user role assignments
3642025-03-21T02:04:26.529ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3652025-03-21T02:04:26.529ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3662025-03-21T02:04:26.536ZINFOnexus (DataLoader): created 2 silo user role assignments file = nexus/db-queries/src/db/datastore/silo_user.rs:446
3672025-03-21T02:04:26.536ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3682025-03-21T02:04:26.536ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3692025-03-21T02:04:26.544ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3702025-03-21T02:04:26.544ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3712025-03-21T02:04:26.555ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3722025-03-21T02:04:26.555ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3732025-03-21T02:04:26.556ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3742025-03-21T02:04:26.557ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
3752025-03-21T02:04:26.557ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3762025-03-21T02:04:26.558ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3772025-03-21T02:04:26.578ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3782025-03-21T02:04:26.578ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3792025-03-21T02:04:26.579ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3802025-03-21T02:04:26.580ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
3812025-03-21T02:04:26.580ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3822025-03-21T02:04:26.580ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3832025-03-21T02:04:26.872ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3842025-03-21T02:04:26.872ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3852025-03-21T02:04:26.881ZINFOnexus: populate complete file = nexus/src/app/mod.rs:545
3862025-03-21T02:04:26.881ZINFOnexus: activating background tasks file = nexus/src/app/mod.rs:554
3872025-03-21T02:04:26.914ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
3882025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
3892025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
3902025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
3912025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
3922025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
3932025-03-21T02:04:26.915ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:71 reason = no blueprint
3942025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
3952025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
3962025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
3972025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
3982025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
3992025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
4002025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4012025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
4022025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
4032025-03-21T02:04:26.915ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector file = nexus/src/app/background/tasks/crdb_node_id_collector.rs:78 reason = no blueprint
4042025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
4052025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
4062025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
4072025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
4082025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
4092025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
4102025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
4112025-03-21T02:04:26.915ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:61 reason = no config nor servers
4122025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
4132025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
4142025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
4152025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
4162025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
4172025-03-21T02:04:26.915ZWARNnexus (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous file = nexus/src/app/background/tasks/blueprint_rendezvous.rs:52 reason = no blueprint
4182025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
4192025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
4202025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
4212025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
4222025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
4232025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
4242025-03-21T02:04:26.915ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:61 reason = no config nor servers
4252025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
4262025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
4272025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4282025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
4292025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
4302025-03-21T02:04:26.915ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
4312025-03-21T02:04:26.917ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
4322025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 1.516906ms iteration = 1 status = {"error":"no blueprint"}
4332025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 1.447666ms iteration = 1 status = {"error":"no blueprint"}
4342025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
4352025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
4362025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
4372025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
4382025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 897.666\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
4392025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
4402025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4412025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
4422025-03-21T02:04:26.918ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
4432025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4442025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
4452025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4462025-03-21T02:04:26.918ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
4472025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
4482025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4492025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 1.018727ms iteration = 1 status = {"error":"no config nor servers"}
4502025-03-21T02:04:26.918ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
4512025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
4522025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
4532025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 991.637\u{b5}s iteration = 1 status = {"error":"no blueprint"}
4542025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
4552025-03-21T02:04:26.918ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
4562025-03-21T02:04:26.919ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
4572025-03-21T02:04:26.923ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
4582025-03-21T02:04:26.923ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
4592025-03-21T02:04:26.923ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4602025-03-21T02:04:26.923ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4612025-03-21T02:04:26.923ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
4622025-03-21T02:04:26.923ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 975, tv_nsec: 550706887 } })
4632025-03-21T02:04:26.924ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4642025-03-21T02:04:26.924ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
4652025-03-21T02:04:26.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4662025-03-21T02:04:26.924ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
4672025-03-21T02:04:26.924ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
4682025-03-21T02:04:26.924ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
4692025-03-21T02:04:26.924ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 01:54:26.924107619 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
4702025-03-21T02:04:26.924ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4712025-03-21T02:04:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
4722025-03-21T02:04:26.924ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
4732025-03-21T02:04:26.924ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
4742025-03-21T02:04:26.924ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
4752025-03-21T02:04:26.924ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
4762025-03-21T02:04:26.924ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 975, tv_nsec: 551594873 } })
4772025-03-21T02:04:26.925ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4782025-03-21T02:04:26.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4792025-03-21T02:04:26.926ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4802025-03-21T02:04:26.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
4812025-03-21T02:04:26.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
4822025-03-21T02:04:26.927ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4832025-03-21T02:04:26.927ZINFOnexus (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external dns_group = external file = nexus/src/app/background/tasks/dns_servers.rs:113
4842025-03-21T02:04:26.927ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 11.616212ms iteration = 1 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
4852025-03-21T02:04:26.927ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
4862025-03-21T02:04:26.927ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:69 reason = no config
4872025-03-21T02:04:26.927ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 23.361\u{b5}s iteration = 2 status = {"error":"no config"}
4882025-03-21T02:04:26.928ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
4892025-03-21T02:04:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4902025-03-21T02:04:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
4912025-03-21T02:04:26.928ZINFOnexus (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal dns_group = internal file = nexus/src/app/background/tasks/dns_servers.rs:113
4922025-03-21T02:04:26.928ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
4932025-03-21T02:04:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4942025-03-21T02:04:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4952025-03-21T02:04:26.928ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 12.988316ms iteration = 1 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
4962025-03-21T02:04:26.928ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
4972025-03-21T02:04:26.928ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:69 reason = no config
4982025-03-21T02:04:26.928ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 4.51\u{b5}s iteration = 2 status = {"error":"no config"}
4992025-03-21T02:04:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5002025-03-21T02:04:26.929ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5012025-03-21T02:04:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5022025-03-21T02:04:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
5032025-03-21T02:04:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
5042025-03-21T02:04:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5052025-03-21T02:04:26.930ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
5062025-03-21T02:04:26.930ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
5072025-03-21T02:04:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5082025-03-21T02:04:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5092025-03-21T02:04:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
5102025-03-21T02:04:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5112025-03-21T02:04:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5122025-03-21T02:04:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
5132025-03-21T02:04:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
5142025-03-21T02:04:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5152025-03-21T02:04:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5162025-03-21T02:04:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
5172025-03-21T02:04:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
5182025-03-21T02:04:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
5192025-03-21T02:04:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
5202025-03-21T02:04:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5212025-03-21T02:04:26.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
5222025-03-21T02:04:26.933ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5232025-03-21T02:04:26.933ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
5242025-03-21T02:04:26.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5252025-03-21T02:04:26.934ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5262025-03-21T02:04:26.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5272025-03-21T02:04:26.936ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5282025-03-21T02:04:26.937ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 21.687068ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
5292025-03-21T02:04:26.939ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5302025-03-21T02:04:26.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5312025-03-21T02:04:26.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5322025-03-21T02:04:26.946ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5332025-03-21T02:04:26.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5342025-03-21T02:04:26.954ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5352025-03-21T02:04:26.956ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5362025-03-21T02:04:26.956ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5372025-03-21T02:04:26.956ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5382025-03-21T02:04:26.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5392025-03-21T02:04:26.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5402025-03-21T02:04:26.958ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5412025-03-21T02:04:26.959ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5422025-03-21T02:04:26.962ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
5432025-03-21T02:04:26.962ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5442025-03-21T02:04:26.962ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5452025-03-21T02:04:26.962ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5462025-03-21T02:04:26.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5472025-03-21T02:04:26.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5482025-03-21T02:04:26.963ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5492025-03-21T02:04:26.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5502025-03-21T02:04:26.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 975, tv_nsec: 613193145 } })
5512025-03-21T02:04:26.986ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
5522025-03-21T02:04:26.986ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
5532025-03-21T02:04:26.986ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 70.553148ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
5542025-03-21T02:04:26.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
5552025-03-21T02:04:26.988ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 73.127543ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
5562025-03-21T02:04:26.990ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 01:54:26.924107619 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
5572025-03-21T02:04:26.990ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 74.483286ms iteration = 1 status = {"expiration":"2025-03-21T01:54:26.924107619Z","pruned":[]}
5582025-03-21T02:04:26.992ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5592025-03-21T02:04:26.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5602025-03-21T02:04:26.996ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5612025-03-21T02:04:26.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5622025-03-21T02:04:26.999ZWARNnexus (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set file = nexus/src/app/background/tasks/blueprint_load.rs:69
5632025-03-21T02:04:26.999ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 83.724738ms iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
5642025-03-21T02:04:27.003ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5652025-03-21T02:04:27.004ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
5662025-03-21T02:04:27.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
5672025-03-21T02:04:27.007ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5682025-03-21T02:04:27.010ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5692025-03-21T02:04:27.011ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
5702025-03-21T02:04:27.011ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 95.590244ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
5712025-03-21T02:04:27.012ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5722025-03-21T02:04:27.012ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5732025-03-21T02:04:27.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5742025-03-21T02:04:27.013ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
5752025-03-21T02:04:27.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5762025-03-21T02:04:27.014ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5772025-03-21T02:04:27.014ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 98.970463ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
5782025-03-21T02:04:27.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5792025-03-21T02:04:27.015ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 100.097683ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
5802025-03-21T02:04:27.016ZINFOnexus (BackgroundTasks): Physical Disk Adoption: Rack not yet initialized background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:80 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
5812025-03-21T02:04:27.016ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 99.970661ms iteration = 1 status = {"error":"rack not yet initialized: 9e10fa12-3248-42fd-baa3-e5de51ba7745"}
5822025-03-21T02:04:27.017ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5832025-03-21T02:04:27.017ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5842025-03-21T02:04:27.018ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5852025-03-21T02:04:27.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5862025-03-21T02:04:27.021ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
5872025-03-21T02:04:27.021ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
5882025-03-21T02:04:27.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
5892025-03-21T02:04:27.022ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
5902025-03-21T02:04:27.022ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
5912025-03-21T02:04:27.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
5922025-03-21T02:04:27.026ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5932025-03-21T02:04:27.027ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5942025-03-21T02:04:27.029ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
5952025-03-21T02:04:27.029ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5962025-03-21T02:04:27.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5972025-03-21T02:04:27.029ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
5982025-03-21T02:04:27.029ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
5992025-03-21T02:04:27.029ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
6002025-03-21T02:04:27.030ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6012025-03-21T02:04:27.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
6022025-03-21T02:04:27.030ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
6032025-03-21T02:04:27.030ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "af2a1399-1c67-4f76-9f7d-eda28fed5574", "content-length": "26", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
6042025-03-21T02:04:27.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
6052025-03-21T02:04:27.032ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6062025-03-21T02:04:27.032ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
6072025-03-21T02:04:27.032ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
6082025-03-21T02:04:27.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
6092025-03-21T02:04:27.033ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 117.930563ms iteration = 1 status = {}
6102025-03-21T02:04:27.034ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6112025-03-21T02:04:27.035ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6122025-03-21T02:04:27.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6132025-03-21T02:04:27.035ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6142025-03-21T02:04:27.036ZWARNnexus (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker file = nexus/src/app/background/tasks/sync_service_zone_nat.rs:95
6152025-03-21T02:04:27.036ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 120.899814ms iteration = 1 status = {"error":"inventory collection is None"}
6162025-03-21T02:04:27.037ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 975, tv_nsec: 664258924 } })
6172025-03-21T02:04:27.037ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
6182025-03-21T02:04:27.052ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6192025-03-21T02:04:27.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6202025-03-21T02:04:27.053ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6212025-03-21T02:04:27.056ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 141.746317ms iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6222025-03-21T02:04:27.057ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
6232025-03-21T02:04:27.057ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
6242025-03-21T02:04:27.057ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
6252025-03-21T02:04:27.058ZWARNnexus (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0 file = nexus/src/app/background/tasks/dns_config.rs:72
6262025-03-21T02:04:27.058ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 142.633773ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
6272025-03-21T02:04:27.058ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6282025-03-21T02:04:27.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6292025-03-21T02:04:27.064ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
6302025-03-21T02:04:27.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
6312025-03-21T02:04:27.074ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 158.464829ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
6322025-03-21T02:04:27.081ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
6332025-03-21T02:04:27.082ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 975, tv_nsec: 709438181 } })
6342025-03-21T02:04:27.087ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6352025-03-21T02:04:27.091ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
6362025-03-21T02:04:27.091ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6372025-03-21T02:04:27.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6382025-03-21T02:04:27.094ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 178.171111ms iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6392025-03-21T02:04:27.103ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
6402025-03-21T02:04:27.103ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
6412025-03-21T02:04:27.103ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 187.393303ms iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
6422025-03-21T02:04:27.104ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6432025-03-21T02:04:27.104ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
6442025-03-21T02:04:27.104ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
6452025-03-21T02:04:27.104ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
6462025-03-21T02:04:27.108ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
6472025-03-21T02:04:27.108ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 192.975529ms iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
6482025-03-21T02:04:27.110ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 195.587105ms iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
6492025-03-21T02:04:27.110ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
6502025-03-21T02:04:27.111ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
6512025-03-21T02:04:27.114ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6522025-03-21T02:04:27.114ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
6532025-03-21T02:04:27.114ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6542025-03-21T02:04:27.114ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "24080546-54d3-4669-8a0f-17d41c493773", "content-length": "26", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
6552025-03-21T02:04:27.114ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6562025-03-21T02:04:27.115ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6572025-03-21T02:04:27.117ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
6582025-03-21T02:04:27.117ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6592025-03-21T02:04:27.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6602025-03-21T02:04:27.119ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6612025-03-21T02:04:27.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6622025-03-21T02:04:27.121ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6632025-03-21T02:04:27.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6642025-03-21T02:04:27.123ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 208.387278ms iteration = 1 status = {}
6652025-03-21T02:04:27.130ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 213.899823ms iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
6662025-03-21T02:04:27.156ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
6672025-03-21T02:04:27.157ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
6682025-03-21T02:04:27.157ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
6692025-03-21T02:04:27.158ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "33b1d1a1-4cd2-4587-b584-d06e20b805cc", "content-length": "476", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
6702025-03-21T02:04:27.159ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
6712025-03-21T02:04:27.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
6722025-03-21T02:04:27.161ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6732025-03-21T02:04:27.163ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
6742025-03-21T02:04:27.163ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
6752025-03-21T02:04:27.163ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
6762025-03-21T02:04:27.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
6772025-03-21T02:04:27.163ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f5b874e-ff5c-45d8-898e-44586e9f50da", "content-length": "1", "date": "Fri, 21 Mar 2025 02:04:27 GMT"} })
6782025-03-21T02:04:27.164ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
6792025-03-21T02:04:27.165ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6c61b83-42f2-4792-b382-18c04b169671", "content-length": "734", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
6802025-03-21T02:04:27.165ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6812025-03-21T02:04:27.165ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6822025-03-21T02:04:27.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6832025-03-21T02:04:27.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6842025-03-21T02:04:27.166ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
6852025-03-21T02:04:27.166ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 251.175633ms iteration = 1 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":0,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":0,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[]}
6862025-03-21T02:04:27.166ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "753b27ca-a207-4141-a9ef-1fa6454947c8", "content-length": "111", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
6872025-03-21T02:04:27.167ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 251.171963ms iteration = 1 status = null
6882025-03-21T02:04:27.167ZWARNnexus (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone file = nexus/src/app/background/tasks/external_endpoints.rs:48
6892025-03-21T02:04:27.167ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 252.405135ms iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
6902025-03-21T02:04:27.167ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
6912025-03-21T02:04:27.168ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b05c99ed-4175-4344-a61f-e052afae7c3b", "content-length": "111", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
6922025-03-21T02:04:27.168ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
6932025-03-21T02:04:27.168ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4542d027-9307-45f8-8679-59e4beb45c93", "content-length": "167", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
6942025-03-21T02:04:27.168ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
6952025-03-21T02:04:27.168ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "390f9d27-1661-436c-ac58-a2edbec35823", "content-length": "167", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
6962025-03-21T02:04:27.168ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
6972025-03-21T02:04:27.169ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f771429e-48ca-4441-8ad0-10c6a10bd968", "content-length": "175", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
6982025-03-21T02:04:27.169ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
6992025-03-21T02:04:27.169ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af7d83c0-2ae4-4676-8616-bd0bf176fe46", "content-length": "175", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7002025-03-21T02:04:27.169ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
7012025-03-21T02:04:27.169ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18a0c93a-445b-4c25-8b34-adee29c20d59", "content-length": "702", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7022025-03-21T02:04:27.169ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
7032025-03-21T02:04:27.169ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f03fcd72-7fcc-4c6f-88a1-2afbbf493733", "content-length": "727", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7042025-03-21T02:04:27.170ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
7052025-03-21T02:04:27.170ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d7d7d28-7fce-4bc7-9c42-dcd1226b2f94", "content-length": "729", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7062025-03-21T02:04:27.170ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
7072025-03-21T02:04:27.170ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e49488c0-1b0e-4c56-936b-9700fa88032a", "content-length": "728", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7082025-03-21T02:04:27.170ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
7092025-03-21T02:04:27.170ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "421c9a82-b3e8-4a22-82d0-b23ce93c62a7", "content-length": "722", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7102025-03-21T02:04:27.170ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
7112025-03-21T02:04:27.171ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49cb1404-6e21-46c7-9e7a-8687fe06b79c", "content-length": "109", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7122025-03-21T02:04:27.171ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
7132025-03-21T02:04:27.171ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "edb2950b-e56c-4e26-ac1a-03e5645c0f62", "content-length": "109", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7142025-03-21T02:04:27.171ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
7152025-03-21T02:04:27.171ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97883314-07d0-4924-8dda-58b8f8614e61", "content-length": "169", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7162025-03-21T02:04:27.171ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
7172025-03-21T02:04:27.171ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34c97d68-9bf6-4369-a27d-db81938de5d1", "content-length": "169", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7182025-03-21T02:04:27.171ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
7192025-03-21T02:04:27.172ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "776cb93f-9bd1-4f32-a9c7-5be9f2f858cb", "content-length": "178", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7202025-03-21T02:04:27.172ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
7212025-03-21T02:04:27.172ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87e9ffe1-b6df-4bb0-853b-d7f74b6f8bde", "content-length": "178", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7222025-03-21T02:04:27.172ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
7232025-03-21T02:04:27.172ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2e480c2-a78d-4abb-b68e-fe56e69f316b", "content-length": "702", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7242025-03-21T02:04:27.172ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
7252025-03-21T02:04:27.172ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8eff0b3a-30b4-4cde-a1d9-a044f50901a9", "content-length": "727", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7262025-03-21T02:04:27.172ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
7272025-03-21T02:04:27.173ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8175514-2819-45c6-9ad7-43ef28656501", "content-length": "729", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7282025-03-21T02:04:27.173ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
7292025-03-21T02:04:27.173ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 257.456172ms iteration = 1 status = {}
7302025-03-21T02:04:27.173ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0cbb21e-21ee-4c67-a9d8-9182b3d87ab4", "content-length": "728", "date": "Fri, 21 Mar 2025 02:04:26 GMT"} })
7312025-03-21T02:04:27.173ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7322025-03-21T02:04:27.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7332025-03-21T02:04:27.173ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
7342025-03-21T02:04:27.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
7352025-03-21T02:04:27.175ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7362025-03-21T02:04:27.176ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
7372025-03-21T02:04:27.176ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7382025-03-21T02:04:27.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7392025-03-21T02:04:27.177ZWARNnexus (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0 file = nexus/src/app/background/tasks/dns_config.rs:72
7402025-03-21T02:04:27.177ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 262.005471ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
7412025-03-21T02:04:27.178ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7422025-03-21T02:04:27.180ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
7432025-03-21T02:04:27.180ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7442025-03-21T02:04:27.180ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7452025-03-21T02:04:27.182ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
7462025-03-21T02:04:27.182ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 266.720413ms iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
7472025-03-21T02:04:27.199ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
7482025-03-21T02:04:27.199ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
7492025-03-21T02:04:27.199ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 283.680358ms iteration = 1 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7502025-03-21T02:04:27.208ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7512025-03-21T02:04:27.214ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
7522025-03-21T02:04:27.214ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7532025-03-21T02:04:27.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7542025-03-21T02:04:27.216ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7552025-03-21T02:04:27.217ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7562025-03-21T02:04:27.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7572025-03-21T02:04:27.218ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
7582025-03-21T02:04:27.218ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
7592025-03-21T02:04:27.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
7602025-03-21T02:04:27.219ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7612025-03-21T02:04:27.220ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
7622025-03-21T02:04:27.220ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 303.896701ms iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
7632025-03-21T02:04:27.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7642025-03-21T02:04:27.221ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7652025-03-21T02:04:27.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7662025-03-21T02:04:27.223ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7672025-03-21T02:04:27.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7682025-03-21T02:04:27.224ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7692025-03-21T02:04:27.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7702025-03-21T02:04:27.226ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7712025-03-21T02:04:27.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7722025-03-21T02:04:27.228ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7732025-03-21T02:04:27.231ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
7742025-03-21T02:04:27.232ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7752025-03-21T02:04:27.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7762025-03-21T02:04:27.233ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7772025-03-21T02:04:27.234ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7782025-03-21T02:04:27.235ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7792025-03-21T02:04:27.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7802025-03-21T02:04:27.237ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7812025-03-21T02:04:27.237ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7822025-03-21T02:04:27.240ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7832025-03-21T02:04:27.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7842025-03-21T02:04:27.241ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7852025-03-21T02:04:27.242ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7862025-03-21T02:04:27.243ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7872025-03-21T02:04:27.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7882025-03-21T02:04:27.244ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7892025-03-21T02:04:27.245ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
7902025-03-21T02:04:27.245ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
7912025-03-21T02:04:27.245ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
7922025-03-21T02:04:27.246ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7932025-03-21T02:04:27.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7942025-03-21T02:04:27.248ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7952025-03-21T02:04:27.248ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
7962025-03-21T02:04:27.248ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7972025-03-21T02:04:27.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7982025-03-21T02:04:27.249ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
7992025-03-21T02:04:27.249ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
8002025-03-21T02:04:27.249ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8012025-03-21T02:04:27.250ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
8022025-03-21T02:04:27.250ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8032025-03-21T02:04:27.251ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8042025-03-21T02:04:27.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8052025-03-21T02:04:27.252ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8062025-03-21T02:04:27.256ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
8072025-03-21T02:04:27.256ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8082025-03-21T02:04:27.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8092025-03-21T02:04:27.431ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8102025-03-21T02:04:27.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8112025-03-21T02:04:27.432ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8122025-03-21T02:04:27.433ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8132025-03-21T02:04:27.435ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8142025-03-21T02:04:27.435ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8152025-03-21T02:04:27.437ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8162025-03-21T02:04:27.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8172025-03-21T02:04:27.441ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8182025-03-21T02:04:27.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8192025-03-21T02:04:27.443ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8202025-03-21T02:04:27.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8212025-03-21T02:04:27.445ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8222025-03-21T02:04:27.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8232025-03-21T02:04:27.446ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8242025-03-21T02:04:27.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8252025-03-21T02:04:27.446ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8262025-03-21T02:04:27.447ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8272025-03-21T02:04:27.449ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
8282025-03-21T02:04:27.449ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8292025-03-21T02:04:27.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8302025-03-21T02:04:27.451ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8312025-03-21T02:04:27.451ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
8322025-03-21T02:04:27.451ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8332025-03-21T02:04:27.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8342025-03-21T02:04:27.453ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8352025-03-21T02:04:27.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8362025-03-21T02:04:27.453ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
8372025-03-21T02:04:27.453ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8382025-03-21T02:04:27.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8392025-03-21T02:04:27.454ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8402025-03-21T02:04:27.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8412025-03-21T02:04:27.455ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8422025-03-21T02:04:27.460ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
8432025-03-21T02:04:27.460ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8442025-03-21T02:04:27.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8452025-03-21T02:04:27.468ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
8462025-03-21T02:04:27.469ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8472025-03-21T02:04:27.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8482025-03-21T02:04:27.471ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8492025-03-21T02:04:27.472ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
8502025-03-21T02:04:27.472ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8512025-03-21T02:04:27.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8522025-03-21T02:04:27.499ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
8532025-03-21T02:04:27.499ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
8542025-03-21T02:04:27.499ZDEBGnexus (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
8552025-03-21T02:04:27.519ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 603.13932ms iteration = 1 status = {}
8562025-03-21T02:04:27.621ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
8572025-03-21T02:04:27.621ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
8582025-03-21T02:04:27.621ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
8592025-03-21T02:04:27.622ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 703.176391ms
8602025-03-21T02:04:27.622ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 706.045432ms iteration = 1 status = {}
8612025-03-21T02:04:27.739ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = c619b06c-12f1-424e-9c3c-2bdf07aa5f54 file = nexus/db-queries/src/db/datastore/inventory.rs:1010
8622025-03-21T02:04:27.739ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = c619b06c-12f1-424e-9c3c-2bdf07aa5f54 time_started = 2025-03-21 02:04:27.153211 UTC
8632025-03-21T02:04:27.739ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 824.654876ms iteration = 1 status = {"collection_id":"c619b06c-12f1-424e-9c3c-2bdf07aa5f54","time_done":"2025-03-21 02:04:27.245619 UTC","time_started":"2025-03-21 02:04:27.153211 UTC"}
8642025-03-21T02:04:27.739ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
8652025-03-21T02:04:27.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
8662025-03-21T02:04:27.739ZWARNnexus (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous file = nexus/src/app/background/tasks/blueprint_rendezvous.rs:52 reason = no blueprint
8672025-03-21T02:04:27.739ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8682025-03-21T02:04:27.739ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 15.541\u{b5}s iteration = 2 status = {"error":"no blueprint"}
8692025-03-21T02:04:27.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8702025-03-21T02:04:27.742ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8712025-03-21T02:04:27.743ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
8722025-03-21T02:04:27.743ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8732025-03-21T02:04:27.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8742025-03-21T02:04:27.745ZINFOnexus (BackgroundTasks): Physical Disk Adoption: Rack not yet initialized background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:80 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
8752025-03-21T02:04:27.745ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 5.987844ms iteration = 2 status = {"error":"rack not yet initialized: 9e10fa12-3248-42fd-baa3-e5de51ba7745"}
8762025-03-21T02:04:27.926ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8772025-03-21T02:04:27.927ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8782025-03-21T02:04:27.928ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8792025-03-21T02:04:27.928ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8802025-03-21T02:04:27.930ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8812025-03-21T02:04:27.932ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 9e10fa12-3248-42fd-baa3-e5de51ba7745, lookup_type: ById(9e10fa12-3248-42fd-baa3-e5de51ba7745) } result = Ok(())
8822025-03-21T02:04:27.932ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, time_created: 2025-03-21T02:04:26.544366Z, time_modified: 2025-03-21T02:04:26.544366Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
8832025-03-21T02:04:29.933ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8842025-03-21T02:04:29.933ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8852025-03-21T02:04:29.935ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8862025-03-21T02:04:29.935ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8872025-03-21T02:04:29.936ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8882025-03-21T02:04:29.939ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 9e10fa12-3248-42fd-baa3-e5de51ba7745, lookup_type: ById(9e10fa12-3248-42fd-baa3-e5de51ba7745) } result = Ok(())
8892025-03-21T02:04:29.939ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, time_created: 2025-03-21T02:04:26.544366Z, time_modified: 2025-03-21T02:04:26.544366Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
8902025-03-21T02:04:31.940ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8912025-03-21T02:04:31.940ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8922025-03-21T02:04:31.941ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8932025-03-21T02:04:31.942ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8942025-03-21T02:04:31.943ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8952025-03-21T02:04:31.945ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 9e10fa12-3248-42fd-baa3-e5de51ba7745, lookup_type: ById(9e10fa12-3248-42fd-baa3-e5de51ba7745) } result = Ok(())
8962025-03-21T02:04:31.945ZINFOnexus: Rack initialized file = nexus/src/app/rack.rs:767
8972025-03-21T02:04:31.945ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
8982025-03-21T02:04:31.946ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
8992025-03-21T02:04:31.947ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9002025-03-21T02:04:31.948ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9012025-03-21T02:04:31.949ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9022025-03-21T02:04:31.949ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9032025-03-21T02:04:31.951ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9042025-03-21T02:04:31.951ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9052025-03-21T02:04:31.952ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9062025-03-21T02:04:31.952ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9072025-03-21T02:04:31.954ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9082025-03-21T02:04:31.954ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9092025-03-21T02:04:31.955ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9102025-03-21T02:04:31.959ZDEBGnexus (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
9112025-03-21T02:04:31.959ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9122025-03-21T02:04:31.959ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9132025-03-21T02:04:31.960ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9142025-03-21T02:04:31.960ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9152025-03-21T02:04:31.961ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9162025-03-21T02:04:31.961ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9172025-03-21T02:04:31.962ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9182025-03-21T02:04:31.966ZDEBGnexus (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
9192025-03-21T02:04:31.966ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9202025-03-21T02:04:31.966ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9212025-03-21T02:04:31.968ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9222025-03-21T02:04:31.969ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9232025-03-21T02:04:31.970ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9242025-03-21T02:04:31.970ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9252025-03-21T02:04:31.971ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9262025-03-21T02:04:31.971ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9272025-03-21T02:04:31.972ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9282025-03-21T02:04:31.973ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9292025-03-21T02:04:31.974ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9302025-03-21T02:04:31.974ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9312025-03-21T02:04:31.975ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9322025-03-21T02:04:31.975ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9332025-03-21T02:04:31.977ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9342025-03-21T02:04:31.980ZDEBGnexus (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
9352025-03-21T02:04:31.980ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9362025-03-21T02:04:31.980ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9372025-03-21T02:04:31.981ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9382025-03-21T02:04:31.981ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9392025-03-21T02:04:31.983ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9402025-03-21T02:04:31.983ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9412025-03-21T02:04:31.986ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9422025-03-21T02:04:31.986ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9432025-03-21T02:04:31.987ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9442025-03-21T02:04:31.988ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9452025-03-21T02:04:31.989ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9462025-03-21T02:04:31.989ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9472025-03-21T02:04:31.990ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9482025-03-21T02:04:31.990ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9492025-03-21T02:04:31.992ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9502025-03-21T02:04:31.995ZDEBGnexus (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
9512025-03-21T02:04:31.995ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9522025-03-21T02:04:31.995ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9532025-03-21T02:04:31.999ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9542025-03-21T02:04:31.999ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9552025-03-21T02:04:32.000ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9562025-03-21T02:04:32.001ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9572025-03-21T02:04:32.002ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9582025-03-21T02:04:32.006ZDEBGnexus (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
9592025-03-21T02:04:32.006ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9602025-03-21T02:04:32.006ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9612025-03-21T02:04:32.010ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9622025-03-21T02:04:32.010ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9632025-03-21T02:04:32.011ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9642025-03-21T02:04:32.012ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9652025-03-21T02:04:32.014ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9662025-03-21T02:04:32.014ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9672025-03-21T02:04:32.016ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9682025-03-21T02:04:32.016ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9692025-03-21T02:04:32.019ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9702025-03-21T02:04:32.019ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9712025-03-21T02:04:32.020ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9722025-03-21T02:04:32.020ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9732025-03-21T02:04:32.022ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9742025-03-21T02:04:32.022ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9752025-03-21T02:04:32.023ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9762025-03-21T02:04:32.027ZDEBGnexus (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
9772025-03-21T02:04:32.027ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9782025-03-21T02:04:32.028ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9792025-03-21T02:04:32.033ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9802025-03-21T02:04:32.033ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9812025-03-21T02:04:32.034ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9822025-03-21T02:04:32.034ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9832025-03-21T02:04:32.036ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9842025-03-21T02:04:32.039ZDEBGnexus (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
9852025-03-21T02:04:32.039ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9862025-03-21T02:04:32.039ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9872025-03-21T02:04:32.048ZDEBGnexus (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 34365a8b-47e5-4623-9ff4-fa54efca49f0, kind: Service { id: 6021c9d9-927c-44a6-8331-6ab48efd9c00 }, name: Name("nexus-6021c9d9-927c-44a6-8331-6ab48efd9c00"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 210])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8d8a5138-cad9-4d17-aa4a-561116e6b6ee, kind: Service { id: e05eb799-8cf0-4702-bd3b-9994ae86118a }, name: Name("nexus-e05eb799-8cf0-4702-bd3b-9994ae86118a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 112])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c4619e25-4546-4262-8518-a5e7bdb39205, kind: Service { id: 4b42f969-b761-4b99-9019-b41841cb0f60 }, name: Name("nexus-4b42f969-b761-4b99-9019-b41841cb0f60"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 128])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 5a5beddc-544e-4e62-bf20-c61d9dde2785, kind: Service { id: bfbad16c-d6a4-4742-9db2-139d724c0771 }, name: Name("external-dns-bfbad16c-d6a4-4742-9db2-139d724c0771"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 221])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bfb6f529-ad0a-4d8f-8e27-90ee0b561301, kind: Service { id: a1777e88-1a9c-49b1-bca1-ec6c5e1991fc }, name: Name("external-dns-a1777e88-1a9c-49b1-bca1-ec6c5e1991fc"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 183, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
9882025-03-21T02:04:32.048ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9892025-03-21T02:04:32.048ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9902025-03-21T02:04:32.050ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9912025-03-21T02:04:32.051ZDEBGnexus (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
9922025-03-21T02:04:32.051ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9932025-03-21T02:04:32.051ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9942025-03-21T02:04:32.053ZDEBGnexus (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
9952025-03-21T02:04:32.056ZDEBGnexus (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 5a5beddc-544e-4e62-bf20-c61d9dde2785, kind: Service { id: bfbad16c-d6a4-4742-9db2-139d724c0771 }, name: Name("external-dns-bfbad16c-d6a4-4742-9db2-139d724c0771"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 221])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bfb6f529-ad0a-4d8f-8e27-90ee0b561301, kind: Service { id: a1777e88-1a9c-49b1-bca1-ec6c5e1991fc }, name: Name("external-dns-a1777e88-1a9c-49b1-bca1-ec6c5e1991fc"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 183, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 34365a8b-47e5-4623-9ff4-fa54efca49f0, kind: Service { id: 6021c9d9-927c-44a6-8331-6ab48efd9c00 }, name: Name("nexus-6021c9d9-927c-44a6-8331-6ab48efd9c00"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 210])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8d8a5138-cad9-4d17-aa4a-561116e6b6ee, kind: Service { id: e05eb799-8cf0-4702-bd3b-9994ae86118a }, name: Name("nexus-e05eb799-8cf0-4702-bd3b-9994ae86118a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 112])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c4619e25-4546-4262-8518-a5e7bdb39205, kind: Service { id: 4b42f969-b761-4b99-9019-b41841cb0f60 }, name: Name("nexus-4b42f969-b761-4b99-9019-b41841cb0f60"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 128])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
9962025-03-21T02:04:32.056ZDEBGnexus (InternalApi): resolved 2 rules for sleds
9972025-03-21T02:04:32.077ZDEBGnexus (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: ad5b28f8-6cad-4221-97f9-e84a46a6381b, time_created: 2025-03-21T02:04:28.969535Z, time_modified: 2025-03-21T02:04:28.969535Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, is_scrimlet: true, serial_number: "buskin", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(56094621696)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
9982025-03-21T02:04:32.077ZDEBGnexus (InternalApi): sending firewall rules to sled agents
9992025-03-21T02:04:32.077ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10002025-03-21T02:04:32.077ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10012025-03-21T02:04:32.080ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10022025-03-21T02:04:32.080ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10032025-03-21T02:04:32.082ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10042025-03-21T02:04:32.084ZDEBGnexus (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
10052025-03-21T02:04:32.114ZDEBGnexus (InternalApi): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
10062025-03-21T02:04:32.114ZDEBGnexus (InternalApi): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "1c93e3f2-7d02-4957-b0e8-5933beb3a9e6", "date": "Fri, 21 Mar 2025 02:04:32 GMT"} })
10072025-03-21T02:04:32.115ZINFOnexus (InternalApi): updated firewall rules on 1 sleds file = nexus/networking/src/firewall_rules.rs:496
10082025-03-21T02:04:32.115ZINFOnexus: plumbed initial IP allowlist file = nexus/src/app/allow_list.rs:161
10092025-03-21T02:04:36.915ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 2 reason = Timeout
10102025-03-21T02:04:36.915ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 2 reason = Timeout
10112025-03-21T02:04:36.915ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10122025-03-21T02:04:36.915ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10132025-03-21T02:04:36.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10142025-03-21T02:04:36.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10152025-03-21T02:04:36.917ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10162025-03-21T02:04:36.920ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10172025-03-21T02:04:36.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10182025-03-21T02:04:36.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10192025-03-21T02:04:36.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10202025-03-21T02:04:36.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10212025-03-21T02:04:36.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10222025-03-21T02:04:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
10232025-03-21T02:04:36.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10242025-03-21T02:04:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10252025-03-21T02:04:36.949ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10262025-03-21T02:04:36.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10272025-03-21T02:04:36.985ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 69.728174ms iteration = 2 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10282025-03-21T02:04:37.019ZINFOnexus (BackgroundTasks): found new target blueprint (first find) background_task = blueprint_loader file = nexus/src/app/background/tasks/blueprint_load.rs:88 target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
10292025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 104.53149ms iteration = 2 status = {"enabled":false,"status":"first target blueprint","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z","time_found":"2025-03-21T02:04:37.019963424Z"}
10302025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 2 reason = Dependency
10312025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 2 reason = Dependency
10322025-03-21T02:04:37.020ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
10332025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.741\u{b5}s iteration = 2 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
10342025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10352025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10362025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10372025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10382025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10392025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10402025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10412025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10422025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10432025-03-21T02:04:37.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10442025-03-21T02:04:37.022ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10452025-03-21T02:04:37.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
10462025-03-21T02:04:37.024ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10472025-03-21T02:04:37.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10482025-03-21T02:04:37.024ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10492025-03-21T02:04:37.026ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
10502025-03-21T02:04:37.026ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10512025-03-21T02:04:37.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10522025-03-21T02:04:37.026ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10532025-03-21T02:04:37.027ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
10542025-03-21T02:04:37.027ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10552025-03-21T02:04:37.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10562025-03-21T02:04:37.028ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10572025-03-21T02:04:37.029ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
10582025-03-21T02:04:37.029ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10592025-03-21T02:04:37.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10602025-03-21T02:04:37.029ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10612025-03-21T02:04:37.031ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
10622025-03-21T02:04:37.031ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10632025-03-21T02:04:37.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10642025-03-21T02:04:37.064ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::3]:32222/node/id
10652025-03-21T02:04:37.096ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::6]:32222/node/id
10662025-03-21T02:04:37.129ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::4]:32222/node/id
10672025-03-21T02:04:37.165ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::7]:32222/node/id
10682025-03-21T02:04:37.198ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::5]:32222/node/id
10692025-03-21T02:04:37.893ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::7]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "73fd2b6b-e57c-4ce3-a4ba-805f792e8b59", "content-length": "64", "date": "Fri, 21 Mar 2025 02:04:37 GMT"} })
10702025-03-21T02:04:37.893ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10712025-03-21T02:04:37.893ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10722025-03-21T02:04:37.895ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10732025-03-21T02:04:37.897ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
10742025-03-21T02:04:37.897ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10752025-03-21T02:04:37.897ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10762025-03-21T02:04:37.924ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::4]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "99e799e4-6d32-4a4d-a60f-003b8aa69069", "content-length": "64", "date": "Fri, 21 Mar 2025 02:04:37 GMT"} })
10772025-03-21T02:04:37.924ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10782025-03-21T02:04:37.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10792025-03-21T02:04:37.926ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10802025-03-21T02:04:37.927ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
10812025-03-21T02:04:37.927ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10822025-03-21T02:04:37.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10832025-03-21T02:04:37.998ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::6]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2bff768-3cca-44f9-8225-dde4e5a2c052", "content-length": "64", "date": "Fri, 21 Mar 2025 02:04:37 GMT"} })
10842025-03-21T02:04:37.998ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10852025-03-21T02:04:37.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10862025-03-21T02:04:38.000ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10872025-03-21T02:04:38.001ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
10882025-03-21T02:04:38.001ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10892025-03-21T02:04:38.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10902025-03-21T02:04:38.012ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::3]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "77cfec9d-688f-413f-a1fc-6c6eb036a8e7", "content-length": "64", "date": "Fri, 21 Mar 2025 02:04:38 GMT"} })
10912025-03-21T02:04:38.012ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10922025-03-21T02:04:38.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10932025-03-21T02:04:38.015ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10942025-03-21T02:04:38.016ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
10952025-03-21T02:04:38.016ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10962025-03-21T02:04:38.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10972025-03-21T02:04:38.036ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::5]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f605b3e6-ef92-46bd-8ec8-5ca52006e00f", "content-length": "64", "date": "Fri, 21 Mar 2025 02:04:38 GMT"} })
10982025-03-21T02:04:38.036ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10992025-03-21T02:04:38.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11002025-03-21T02:04:38.038ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11012025-03-21T02:04:38.039ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
11022025-03-21T02:04:38.039ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11032025-03-21T02:04:38.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11042025-03-21T02:04:38.051ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 1.031208512s iteration = 2 status = {"nsuccess":5}
11052025-03-21T02:04:46.915ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 3 reason = Timeout
11062025-03-21T02:04:46.915ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11072025-03-21T02:04:46.915ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 3 reason = Timeout
11082025-03-21T02:04:46.915ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11092025-03-21T02:04:46.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11102025-03-21T02:04:46.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11112025-03-21T02:04:46.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11122025-03-21T02:04:46.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11132025-03-21T02:04:46.917ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11142025-03-21T02:04:46.919ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11152025-03-21T02:04:46.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11162025-03-21T02:04:46.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11172025-03-21T02:04:46.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11182025-03-21T02:04:46.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11192025-03-21T02:04:46.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11202025-03-21T02:04:46.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
11212025-03-21T02:04:46.933ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11222025-03-21T02:04:46.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11232025-03-21T02:04:46.944ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
11242025-03-21T02:04:46.944ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 29.323981ms iteration = 3 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
11252025-03-21T02:04:46.948ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.909802ms iteration = 3 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
11262025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 4 reason = Timeout
11272025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11282025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
11292025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
11302025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 2 reason = Timeout
11312025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
11322025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
11332025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
11342025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 2 reason = Timeout
11352025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11362025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
11372025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
11382025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11392025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 4 reason = Timeout
11402025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
11412025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
11422025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
11432025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
11442025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11452025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 2 reason = Timeout
11462025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11472025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
11482025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
11492025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11502025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
11512025-03-21T02:04:56.915ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
11522025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11532025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 2 reason = Timeout
11542025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
11552025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11562025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11572025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
11582025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
11592025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 2 reason = Timeout
11602025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
11612025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 2 reason = Timeout
11622025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
11632025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 2 reason = Timeout
11642025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
11652025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11662025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11672025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11682025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
11692025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11702025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
11712025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
11722025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
11732025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
11742025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
11752025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11762025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11772025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11782025-03-21T02:04:56.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11792025-03-21T02:04:56.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
11802025-03-21T02:04:56.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
11812025-03-21T02:04:56.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11822025-03-21T02:04:56.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11832025-03-21T02:04:56.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11842025-03-21T02:04:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11852025-03-21T02:04:56.922ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11862025-03-21T02:04:56.922ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 7.048383ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
11872025-03-21T02:04:56.923ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1005, tv_nsec: 549997936 } })
11882025-03-21T02:04:56.923ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
11892025-03-21T02:04:56.924ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11902025-03-21T02:04:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
11912025-03-21T02:04:56.925ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11922025-03-21T02:04:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11932025-03-21T02:04:56.926ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
11942025-03-21T02:04:56.926ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
11952025-03-21T02:04:56.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
11962025-03-21T02:04:56.927ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1005, tv_nsec: 554596196 } })
11972025-03-21T02:04:56.928ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11982025-03-21T02:04:56.928ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
11992025-03-21T02:04:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
12002025-03-21T02:04:56.930ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12012025-03-21T02:04:56.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12022025-03-21T02:04:56.931ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12032025-03-21T02:04:56.932ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 16.619419ms iteration = 2 status = {"errors":[],"garbage_collect_requested":[]}
12042025-03-21T02:04:56.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12052025-03-21T02:04:56.933ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12062025-03-21T02:04:56.933ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12072025-03-21T02:04:56.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12082025-03-21T02:04:56.933ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12092025-03-21T02:04:56.934ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12102025-03-21T02:04:56.935ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12112025-03-21T02:04:56.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12122025-03-21T02:04:56.936ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12132025-03-21T02:04:56.936ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
12142025-03-21T02:04:56.936ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12152025-03-21T02:04:56.936ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
12162025-03-21T02:04:56.936ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12172025-03-21T02:04:56.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12182025-03-21T02:04:56.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12192025-03-21T02:04:56.937ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12202025-03-21T02:04:56.937ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12212025-03-21T02:04:56.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12222025-03-21T02:04:56.938ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12232025-03-21T02:04:56.938ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12242025-03-21T02:04:56.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12252025-03-21T02:04:56.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12262025-03-21T02:04:56.946ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12272025-03-21T02:04:56.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12282025-03-21T02:04:56.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12292025-03-21T02:04:56.947ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1005, tv_nsec: 573935452 } })
12302025-03-21T02:04:56.949ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
12312025-03-21T02:04:56.960ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12322025-03-21T02:04:56.960ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
12332025-03-21T02:04:56.960ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
12342025-03-21T02:04:56.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12352025-03-21T02:04:56.962ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
12362025-03-21T02:04:56.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
12372025-03-21T02:04:56.966ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
12382025-03-21T02:04:56.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
12392025-03-21T02:04:56.968ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 53.109314ms iteration = 2 status = {"errors":[],"requests_created_ok":[]}
12402025-03-21T02:04:56.971ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 55.851482ms iteration = 2 status = {"errors":[],"finish_invoked_ok":[]}
12412025-03-21T02:04:56.972ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12422025-03-21T02:04:56.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12432025-03-21T02:04:56.978ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 63.705218ms iteration = 4 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12442025-03-21T02:04:56.979ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12452025-03-21T02:04:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12462025-03-21T02:04:56.982ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
12472025-03-21T02:04:56.982ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 66.748232ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
12482025-03-21T02:04:57.039ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12492025-03-21T02:04:57.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12502025-03-21T02:04:57.070ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
12512025-03-21T02:04:57.070ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
12522025-03-21T02:04:57.072ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12532025-03-21T02:04:57.073ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
12542025-03-21T02:04:57.073ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
12552025-03-21T02:04:57.074ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cf90624-ee2a-475f-99fc-5574a85bd966", "content-length": "26", "date": "Fri, 21 Mar 2025 02:04:57 GMT"} })
12562025-03-21T02:04:57.074ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
12572025-03-21T02:04:57.075ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
12582025-03-21T02:04:57.075ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12592025-03-21T02:04:57.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12602025-03-21T02:04:57.075ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12612025-03-21T02:04:57.075ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
12622025-03-21T02:04:57.078ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
12632025-03-21T02:04:57.078ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12642025-03-21T02:04:57.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12652025-03-21T02:04:57.107ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12662025-03-21T02:04:57.108ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 193.257155ms iteration = 2 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
12672025-03-21T02:04:57.110ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
12682025-03-21T02:04:57.110ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12692025-03-21T02:04:57.110ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12702025-03-21T02:04:57.111ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12712025-03-21T02:04:57.111ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 195.89369ms iteration = 2 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
12722025-03-21T02:04:57.111ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12732025-03-21T02:04:57.116ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
12742025-03-21T02:04:57.116ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 201.417287ms iteration = 2 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
12752025-03-21T02:04:57.122ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12762025-03-21T02:04:57.122ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
12772025-03-21T02:04:57.122ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
12782025-03-21T02:04:57.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12792025-03-21T02:04:57.123ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
12802025-03-21T02:04:57.123ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 208.5639ms iteration = 4 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
12812025-03-21T02:04:57.125ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
12822025-03-21T02:04:57.125ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
12832025-03-21T02:04:57.125ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "902ee0e6-e1e1-4790-8100-843096024d7c", "content-length": "26", "date": "Fri, 21 Mar 2025 02:04:57 GMT"} })
12842025-03-21T02:04:57.125ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
12852025-03-21T02:04:57.126ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
12862025-03-21T02:04:57.127ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
12872025-03-21T02:04:57.127ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
12882025-03-21T02:04:57.128ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e3988cc-cacb-4892-95b7-b7a84d8b16d5", "content-length": "26", "date": "Fri, 21 Mar 2025 02:04:57 GMT"} })
12892025-03-21T02:04:57.128ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
12902025-03-21T02:04:57.129ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
12912025-03-21T02:04:57.149ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 234.044614ms iteration = 2 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
12922025-03-21T02:04:57.158ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12932025-03-21T02:04:57.158ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
12942025-03-21T02:04:57.158ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 242.753956ms iteration = 2 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
12952025-03-21T02:04:57.187ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12962025-03-21T02:04:57.188ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12972025-03-21T02:04:57.190ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12982025-03-21T02:04:57.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12992025-03-21T02:04:57.192ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13002025-03-21T02:04:57.193ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13012025-03-21T02:04:57.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13022025-03-21T02:04:57.196ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13032025-03-21T02:04:57.196ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
13042025-03-21T02:04:57.197ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
13052025-03-21T02:04:57.197ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13062025-03-21T02:04:57.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13072025-03-21T02:04:57.198ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13082025-03-21T02:04:57.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13092025-03-21T02:04:57.199ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13102025-03-21T02:04:57.200ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 284.847329ms iteration = 3 status = {"physical_disks_added":0}
13112025-03-21T02:04:57.201ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
13122025-03-21T02:04:57.201ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13132025-03-21T02:04:57.201ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13142025-03-21T02:04:57.202ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 287.564917ms iteration = 2 status = 0
13152025-03-21T02:04:57.213ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
13162025-03-21T02:04:57.214ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7c22c01-628d-4f17-bdf5-675183f8f73b", "content-length": "1", "date": "Fri, 21 Mar 2025 02:04:56 GMT"} })
13172025-03-21T02:04:57.214ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 298.906814ms iteration = 2 status = {}
13182025-03-21T02:04:57.215ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
13192025-03-21T02:04:57.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
13202025-03-21T02:04:57.217ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 302.627419ms iteration = 2 status = null
13212025-03-21T02:04:57.234ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cc191eb-8ace-41a2-b035-23bc28b58151", "content-length": "372", "date": "Fri, 21 Mar 2025 02:04:57 GMT"} })
13222025-03-21T02:04:57.239ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13232025-03-21T02:04:57.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13242025-03-21T02:04:57.252ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
13252025-03-21T02:04:57.255ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "44a4ec76-832d-4fe1-a11b-a0f99565bbb3", "content-length": "691", "date": "Fri, 21 Mar 2025 02:04:57 GMT"} })
13262025-03-21T02:04:57.257ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
13272025-03-21T02:04:57.257ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
13282025-03-21T02:04:57.257ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
13292025-03-21T02:04:57.259ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 343.699153ms iteration = 2 status = {}
13302025-03-21T02:04:57.275ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13312025-03-21T02:04:57.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13322025-03-21T02:04:57.281ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
13332025-03-21T02:04:57.281ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13342025-03-21T02:04:57.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13352025-03-21T02:04:57.296ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
13362025-03-21T02:04:57.296ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
13372025-03-21T02:04:57.296ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "10d249ca-33a6-4dcc-b633-57a71115fda7", "date": "Fri, 21 Mar 2025 02:04:57 GMT"} })
13382025-03-21T02:04:57.299ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13392025-03-21T02:04:57.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13402025-03-21T02:04:57.301ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13412025-03-21T02:04:57.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13422025-03-21T02:04:57.305ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13432025-03-21T02:04:57.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13442025-03-21T02:04:57.309ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
13452025-03-21T02:04:57.309ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
13462025-03-21T02:04:57.311ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "33dcf395-3aef-450c-be02-d785e91b4b89", "date": "Fri, 21 Mar 2025 02:04:57 GMT"} })
13472025-03-21T02:04:57.313ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 397.827216ms iteration = 2 status = {}
13482025-03-21T02:04:57.426ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
13492025-03-21T02:04:57.427ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "15e082a0-6c30-466a-9c68-1ea387724ceb", "content-length": "134", "date": "Fri, 21 Mar 2025 02:04:56 GMT"} })
13502025-03-21T02:04:57.427ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
13512025-03-21T02:04:57.427ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
13522025-03-21T02:04:57.429ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
13532025-03-21T02:04:57.429ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
13542025-03-21T02:04:57.431ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13552025-03-21T02:04:57.431ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13562025-03-21T02:04:57.431ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13572025-03-21T02:04:57.431ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
13582025-03-21T02:04:57.434ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13592025-03-21T02:04:57.434ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
13602025-03-21T02:04:57.436ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
13612025-03-21T02:04:57.436ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13622025-03-21T02:04:57.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13632025-03-21T02:04:57.438ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
13642025-03-21T02:04:57.440ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "88784bf1-1231-4b38-83e6-35869058a0ca", "content-length": "2", "date": "Fri, 21 Mar 2025 02:04:56 GMT"} })
13652025-03-21T02:04:57.443ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
13662025-03-21T02:04:57.443ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "dda12a42-22ce-4492-aeb3-d2e79a24636c", "content-length": "52", "date": "Fri, 21 Mar 2025 02:04:56 GMT"} })
13672025-03-21T02:04:57.445ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
13682025-03-21T02:04:57.445ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
13692025-03-21T02:04:57.543ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "35389b11-21f0-46e4-85e5-0d9018b7d6ee", "date": "Fri, 21 Mar 2025 02:04:57 GMT"} })
13702025-03-21T02:04:57.543ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
13712025-03-21T02:04:57.543ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
13722025-03-21T02:04:57.543ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "68f05150-63ca-4116-ba72-c12b11beb914", "content-length": "682", "date": "Fri, 21 Mar 2025 02:04:57 GMT"} })
13732025-03-21T02:04:57.543ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13742025-03-21T02:04:57.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13752025-03-21T02:04:57.549ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13762025-03-21T02:04:57.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13772025-03-21T02:04:57.552ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13782025-03-21T02:04:57.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13792025-03-21T02:04:57.555ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13802025-03-21T02:04:57.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13812025-03-21T02:04:57.569ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 654.401013ms iteration = 2 status = {}
13822025-03-21T02:05:06.915ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 5 reason = Timeout
13832025-03-21T02:05:06.915ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 5 reason = Timeout
13842025-03-21T02:05:06.915ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13852025-03-21T02:05:06.915ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13862025-03-21T02:05:06.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13872025-03-21T02:05:06.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13882025-03-21T02:05:06.917ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13892025-03-21T02:05:06.919ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13902025-03-21T02:05:06.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13912025-03-21T02:05:06.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13922025-03-21T02:05:06.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13932025-03-21T02:05:06.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13942025-03-21T02:05:06.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13952025-03-21T02:05:06.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
13962025-03-21T02:05:06.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13972025-03-21T02:05:06.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13982025-03-21T02:05:06.935ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
13992025-03-21T02:05:06.935ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.163961ms iteration = 5 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
14002025-03-21T02:05:06.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14012025-03-21T02:05:06.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14022025-03-21T02:05:06.971ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 56.692107ms iteration = 5 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
14032025-03-21T02:05:16.915ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 6 reason = Timeout
14042025-03-21T02:05:16.915ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 6 reason = Timeout
14052025-03-21T02:05:16.915ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14062025-03-21T02:05:16.915ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14072025-03-21T02:05:16.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14082025-03-21T02:05:16.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14092025-03-21T02:05:16.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14102025-03-21T02:05:16.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14112025-03-21T02:05:16.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14122025-03-21T02:05:16.920ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
14132025-03-21T02:05:16.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14142025-03-21T02:05:16.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14152025-03-21T02:05:16.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14162025-03-21T02:05:16.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14172025-03-21T02:05:16.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14182025-03-21T02:05:16.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
14192025-03-21T02:05:16.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14202025-03-21T02:05:16.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14212025-03-21T02:05:16.938ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
14222025-03-21T02:05:16.938ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.828097ms iteration = 6 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
14232025-03-21T02:05:16.950ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.288077ms iteration = 6 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
14242025-03-21T02:05:26.915ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 3 reason = Timeout
14252025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 3 reason = Timeout
14262025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Timeout
14272025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
14282025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 2 reason = Timeout
14292025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
14302025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14312025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
14322025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 3 reason = Timeout
14332025-03-21T02:05:26.916ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
14342025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 2 reason = Timeout
14352025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 21.7\u{b5}s iteration = 3 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
14362025-03-21T02:05:26.916ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 01:55:26.916024781 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
14372025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Timeout
14382025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 3 reason = Timeout
14392025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
14402025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
14412025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 3 reason = Timeout
14422025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
14432025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
14442025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Timeout
14452025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Timeout
14462025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 4 reason = Timeout
14472025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Timeout
14482025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
14492025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
14502025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14512025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 7 reason = Timeout
14522025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14532025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 7 reason = Timeout
14542025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 2 reason = Timeout
14552025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 2 reason = Timeout
14562025-03-21T02:05:26.916ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
14572025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14582025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
14592025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 3 reason = Timeout
14602025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
14612025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 3 reason = Timeout
14622025-03-21T02:05:26.916ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
14632025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
14642025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14652025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 3 reason = Timeout
14662025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14672025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 3 reason = Timeout
14682025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
14692025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 3 reason = Timeout
14702025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
14712025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 3 reason = Timeout
14722025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
14732025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 3 reason = Timeout
14742025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
14752025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 2 reason = Timeout
14762025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
14772025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 3 reason = Timeout
14782025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
14792025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
14802025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 3 reason = Timeout
14812025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
14822025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Timeout
14832025-03-21T02:05:26.916ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:69 reason = no config
14842025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 12.34\u{b5}s iteration = 3 status = {"error":"no config"}
14852025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Timeout
14862025-03-21T02:05:26.916ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:69 reason = no config
14872025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 10.83\u{b5}s iteration = 3 status = {"error":"no config"}
14882025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 3 reason = Timeout
14892025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 3 reason = Timeout
14902025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 3 reason = Timeout
14912025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14922025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14932025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
14942025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14952025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
14962025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
14972025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
14982025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
14992025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
15002025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
15012025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15022025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
15032025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
15042025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
15052025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15062025-03-21T02:05:26.916ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1035, tv_nsec: 542707823 } })
15072025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15082025-03-21T02:05:26.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
15092025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15102025-03-21T02:05:26.917ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1035, tv_nsec: 542814465 } })
15112025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
15122025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15132025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15142025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
15152025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
15162025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
15172025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
15182025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15192025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15202025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
15212025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
15222025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
15232025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.346274ms iteration = 2 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
15242025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
15252025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.533297ms iteration = 2 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
15262025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
15272025-03-21T02:05:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
15282025-03-21T02:05:26.919ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 3.46211ms iteration = 3 status = {"errors":[],"garbage_collect_requested":[]}
15292025-03-21T02:05:26.920ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15302025-03-21T02:05:26.920ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.517308ms iteration = 3 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
15312025-03-21T02:05:26.921ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 5.366484ms iteration = 3 status = {"errors":[],"finish_invoked_ok":[]}
15322025-03-21T02:05:26.921ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 5.586487ms iteration = 2 status = {"errors":[],"found_port_ok":[]}
15332025-03-21T02:05:26.922ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
15342025-03-21T02:05:26.922ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
15352025-03-21T02:05:26.922ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1035, tv_nsec: 548437243 } })
15362025-03-21T02:05:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
15372025-03-21T02:05:26.922ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
15382025-03-21T02:05:26.925ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15392025-03-21T02:05:26.925ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 01:55:26.916024781 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
15402025-03-21T02:05:26.925ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 9.341803ms iteration = 2 status = {"expiration":"2025-03-21T01:55:26.916024781Z","pruned":[]}
15412025-03-21T02:05:26.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15422025-03-21T02:05:26.926ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15432025-03-21T02:05:26.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
15442025-03-21T02:05:26.927ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
15452025-03-21T02:05:26.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
15462025-03-21T02:05:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
15472025-03-21T02:05:26.929ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15482025-03-21T02:05:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15492025-03-21T02:05:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15502025-03-21T02:05:26.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15512025-03-21T02:05:26.930ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 13.992474ms iteration = 3 status = {"errors":[],"requests_created_ok":[]}
15522025-03-21T02:05:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15532025-03-21T02:05:26.933ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15542025-03-21T02:05:26.933ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
15552025-03-21T02:05:26.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
15562025-03-21T02:05:26.935ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15572025-03-21T02:05:26.935ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15582025-03-21T02:05:26.936ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
15592025-03-21T02:05:26.936ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15602025-03-21T02:05:26.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15612025-03-21T02:05:26.937ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
15622025-03-21T02:05:26.937ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
15632025-03-21T02:05:26.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
15642025-03-21T02:05:26.937ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
15652025-03-21T02:05:26.937ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15662025-03-21T02:05:26.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15672025-03-21T02:05:26.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15682025-03-21T02:05:26.939ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1035, tv_nsec: 565791635 } })
15692025-03-21T02:05:26.940ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
15702025-03-21T02:05:26.942ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
15712025-03-21T02:05:26.942ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15722025-03-21T02:05:26.942ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
15732025-03-21T02:05:26.942ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
15742025-03-21T02:05:26.942ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
15752025-03-21T02:05:26.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15762025-03-21T02:05:26.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
15772025-03-21T02:05:26.943ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
15782025-03-21T02:05:26.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
15792025-03-21T02:05:26.952ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15802025-03-21T02:05:26.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15812025-03-21T02:05:26.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15822025-03-21T02:05:26.954ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15832025-03-21T02:05:26.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15842025-03-21T02:05:26.956ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
15852025-03-21T02:05:26.957ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1035, tv_nsec: 583486743 } })
15862025-03-21T02:05:26.958ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
15872025-03-21T02:05:26.958ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15882025-03-21T02:05:26.963ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
15892025-03-21T02:05:26.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15902025-03-21T02:05:26.963ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
15912025-03-21T02:05:26.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
15922025-03-21T02:05:26.966ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
15932025-03-21T02:05:26.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15942025-03-21T02:05:26.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15952025-03-21T02:05:26.978ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15962025-03-21T02:05:26.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15972025-03-21T02:05:26.979ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
15982025-03-21T02:05:26.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
15992025-03-21T02:05:26.980ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16002025-03-21T02:05:26.983ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
16012025-03-21T02:05:26.984ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 68.493542ms iteration = 7 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16022025-03-21T02:05:26.985ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
16032025-03-21T02:05:26.985ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
16042025-03-21T02:05:26.985ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 68.811358ms iteration = 2 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
16052025-03-21T02:05:26.986ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
16062025-03-21T02:05:26.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
16072025-03-21T02:05:26.992ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
16082025-03-21T02:05:26.992ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16092025-03-21T02:05:26.992ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "32ed9480-dcb7-40c8-970f-2103eaffe7ef", "content-length": "26", "date": "Fri, 21 Mar 2025 02:05:26 GMT"} })
16102025-03-21T02:05:26.992ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16112025-03-21T02:05:26.993ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
16122025-03-21T02:05:26.997ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16132025-03-21T02:05:27.000ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
16142025-03-21T02:05:27.000ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16152025-03-21T02:05:27.000ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
16162025-03-21T02:05:27.000ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 84.346198ms iteration = 7 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
16172025-03-21T02:05:27.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16182025-03-21T02:05:27.001ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
16192025-03-21T02:05:27.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
16202025-03-21T02:05:27.002ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16212025-03-21T02:05:27.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16222025-03-21T02:05:27.004ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 88.245546ms iteration = 3 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
16232025-03-21T02:05:27.006ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
16242025-03-21T02:05:27.006ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
16252025-03-21T02:05:27.006ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
16262025-03-21T02:05:27.006ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16272025-03-21T02:05:27.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16282025-03-21T02:05:27.008ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16292025-03-21T02:05:27.009ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
16302025-03-21T02:05:27.009ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
16312025-03-21T02:05:27.009ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 93.143891ms iteration = 2 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
16322025-03-21T02:05:27.012ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16332025-03-21T02:05:27.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16342025-03-21T02:05:27.012ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
16352025-03-21T02:05:27.012ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16362025-03-21T02:05:27.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16372025-03-21T02:05:27.013ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16382025-03-21T02:05:27.014ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
16392025-03-21T02:05:27.014ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 98.469534ms iteration = 3 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
16402025-03-21T02:05:27.016ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
16412025-03-21T02:05:27.016ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16422025-03-21T02:05:27.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16432025-03-21T02:05:27.017ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
16442025-03-21T02:05:27.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
16452025-03-21T02:05:27.020ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16462025-03-21T02:05:27.020ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
16472025-03-21T02:05:27.020ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16482025-03-21T02:05:27.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16492025-03-21T02:05:27.023ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16502025-03-21T02:05:27.023ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
16512025-03-21T02:05:27.023ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 107.575673ms iteration = 3 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
16522025-03-21T02:05:27.032ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
16532025-03-21T02:05:27.032ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
16542025-03-21T02:05:27.032ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
16552025-03-21T02:05:27.032ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16562025-03-21T02:05:27.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16572025-03-21T02:05:27.034ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16582025-03-21T02:05:27.035ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
16592025-03-21T02:05:27.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
16602025-03-21T02:05:27.038ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
16612025-03-21T02:05:27.038ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16622025-03-21T02:05:27.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16632025-03-21T02:05:27.040ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 124.263694ms iteration = 3 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
16642025-03-21T02:05:27.044ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
16652025-03-21T02:05:27.044ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16662025-03-21T02:05:27.044ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5991f9b0-af95-4214-b3ec-307a8b5f27ea", "content-length": "26", "date": "Fri, 21 Mar 2025 02:05:26 GMT"} })
16672025-03-21T02:05:27.044ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16682025-03-21T02:05:27.045ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
16692025-03-21T02:05:27.045ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
16702025-03-21T02:05:27.045ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16712025-03-21T02:05:27.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16722025-03-21T02:05:27.046ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
16732025-03-21T02:05:27.053ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
16742025-03-21T02:05:27.053ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 136.894684ms iteration = 3 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
16752025-03-21T02:05:27.059ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16762025-03-21T02:05:27.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16772025-03-21T02:05:27.060ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
16782025-03-21T02:05:27.060ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16792025-03-21T02:05:27.060ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16802025-03-21T02:05:27.061ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd1a9bb1-f221-4470-94b6-dff9cee66453", "content-length": "26", "date": "Fri, 21 Mar 2025 02:05:26 GMT"} })
16812025-03-21T02:05:27.061ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16822025-03-21T02:05:27.061ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16832025-03-21T02:05:27.062ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
16842025-03-21T02:05:27.062ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16852025-03-21T02:05:27.062ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
16862025-03-21T02:05:27.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16872025-03-21T02:05:27.063ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
16882025-03-21T02:05:27.063ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16892025-03-21T02:05:27.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16902025-03-21T02:05:27.077ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 161.696884ms iteration = 3 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
16912025-03-21T02:05:27.080ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
16922025-03-21T02:05:27.080ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
16932025-03-21T02:05:27.080ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16942025-03-21T02:05:27.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16952025-03-21T02:05:27.081ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16962025-03-21T02:05:27.084ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
16972025-03-21T02:05:27.084ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16982025-03-21T02:05:27.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16992025-03-21T02:05:27.085ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
17002025-03-21T02:05:27.086ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6324ce3-1e59-4f5c-940b-43b151169730", "content-length": "1", "date": "Fri, 21 Mar 2025 02:05:26 GMT"} })
17012025-03-21T02:05:27.087ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
17022025-03-21T02:05:27.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
17032025-03-21T02:05:27.091ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17042025-03-21T02:05:27.091ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
17052025-03-21T02:05:27.093ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
17062025-03-21T02:05:27.093ZINFOnexus (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal file = nexus/src/app/background/tasks/dns_config.rs:87 generation = 1
17072025-03-21T02:05:27.093ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 177.714634ms iteration = 2 status = {"generation":1}
17082025-03-21T02:05:27.093ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 4 reason = Dependency
17092025-03-21T02:05:27.094ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
17102025-03-21T02:05:27.094ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17112025-03-21T02:05:27.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17122025-03-21T02:05:27.127ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
17132025-03-21T02:05:27.127ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 211.841027ms iteration = 2 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
17142025-03-21T02:05:27.133ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17152025-03-21T02:05:27.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17162025-03-21T02:05:27.139ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17172025-03-21T02:05:27.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17182025-03-21T02:05:27.183ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 267.555788ms iteration = 4 status = {"physical_disks_added":0}
17192025-03-21T02:05:27.199ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
17202025-03-21T02:05:27.199ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
17212025-03-21T02:05:27.201ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 285.327817ms iteration = 3 status = null
17222025-03-21T02:05:27.202ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17232025-03-21T02:05:27.203ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
17242025-03-21T02:05:27.203ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
17252025-03-21T02:05:27.203ZINFOnexus (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external file = nexus/src/app/background/tasks/dns_config.rs:87 generation = 2
17262025-03-21T02:05:27.203ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 286.932625ms iteration = 2 status = {"generation":2}
17272025-03-21T02:05:27.203ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 4 reason = Dependency
17282025-03-21T02:05:27.203ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17292025-03-21T02:05:27.204ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
17302025-03-21T02:05:27.204ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17312025-03-21T02:05:27.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17322025-03-21T02:05:27.205ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
17332025-03-21T02:05:27.205ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17342025-03-21T02:05:27.205ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17352025-03-21T02:05:27.205ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17362025-03-21T02:05:27.205ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17372025-03-21T02:05:27.207ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17382025-03-21T02:05:27.208ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17392025-03-21T02:05:27.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17402025-03-21T02:05:27.209ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
17412025-03-21T02:05:27.209ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17422025-03-21T02:05:27.209ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17432025-03-21T02:05:27.210ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17442025-03-21T02:05:27.212ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
17452025-03-21T02:05:27.212ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17462025-03-21T02:05:27.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17472025-03-21T02:05:27.215ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 298.884783ms iteration = 3 status = 0
17482025-03-21T02:05:27.220ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc4a962d-fc28-4741-8e74-ab6a4a459af0", "content-length": "372", "date": "Fri, 21 Mar 2025 02:05:27 GMT"} })
17492025-03-21T02:05:27.220ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17502025-03-21T02:05:27.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17512025-03-21T02:05:27.239ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
17522025-03-21T02:05:27.239ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
17532025-03-21T02:05:27.241ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "7d175dcc-12bb-4f6e-b0cd-81776d8f1d58", "date": "Fri, 21 Mar 2025 02:05:27 GMT"} })
17542025-03-21T02:05:27.241ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17552025-03-21T02:05:27.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17562025-03-21T02:05:27.244ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17572025-03-21T02:05:27.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17582025-03-21T02:05:27.247ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17592025-03-21T02:05:27.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17602025-03-21T02:05:27.250ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
17612025-03-21T02:05:27.250ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
17622025-03-21T02:05:27.251ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "ac2a80f3-ddd0-4a76-aa94-81d23d7a408f", "date": "Fri, 21 Mar 2025 02:05:27 GMT"} })
17632025-03-21T02:05:27.252ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 336.639641ms iteration = 3 status = {}
17642025-03-21T02:05:27.257ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 341.552146ms iteration = 3 status = {}
17652025-03-21T02:05:27.267ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:77
17662025-03-21T02:05:27.267ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 351.814175ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
17672025-03-21T02:05:27.294ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
17682025-03-21T02:05:27.319ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
17692025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = / versions = all
17702025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /access versions = all
17712025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /assets/{path} versions = all
17722025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /device/auth versions = all
17732025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /device/confirm versions = all
17742025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /device/success versions = all
17752025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /device/token versions = all
17762025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /device/verify versions = all
17772025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /experimental/v1/probes versions = all
17782025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /experimental/v1/probes versions = all
17792025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /experimental/v1/probes/{probe} versions = all
17802025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /experimental/v1/probes/{probe} versions = all
17812025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /experimental/v1/system/support-bundles versions = all
17822025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /experimental/v1/system/support-bundles versions = all
17832025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
17842025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
17852025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
17862025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
17872025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
17882025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
17892025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
17902025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /images versions = all
17912025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /login versions = all
17922025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /login/{silo_name}/local versions = all
17932025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
17942025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
17952025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
17962025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /lookup/{path} versions = all
17972025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /projects/{path} versions = all
17982025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /projects-new versions = all
17992025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /settings/{path} versions = all
18002025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /system/{path} versions = all
18012025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /utilization versions = all
18022025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/affinity-groups versions = all
18032025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/affinity-groups versions = all
18042025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
18052025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
18062025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
18072025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
18082025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
18092025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
18102025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
18112025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/anti-affinity-groups versions = all
18122025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/anti-affinity-groups versions = all
18132025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
18142025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
18152025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
18162025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
18172025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
18182025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
18192025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
18202025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/certificates versions = all
18212025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/certificates versions = all
18222025-03-21T02:05:27.319ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/certificates/{certificate} versions = all
18232025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/certificates/{certificate} versions = all
18242025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/disks versions = all
18252025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/disks versions = all
18262025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/disks/{disk} versions = all
18272025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/disks/{disk} versions = all
18282025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/disks/{disk}/bulk-write versions = all
18292025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
18302025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
18312025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/disks/{disk}/finalize versions = all
18322025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
18332025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/floating-ips versions = all
18342025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/floating-ips versions = all
18352025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
18362025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/floating-ips/{floating_ip} versions = all
18372025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
18382025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
18392025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
18402025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/groups versions = all
18412025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/groups/{group_id} versions = all
18422025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/images versions = all
18432025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/images versions = all
18442025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/images/{image} versions = all
18452025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/images/{image} versions = all
18462025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/images/{image}/demote versions = all
18472025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/images/{image}/promote versions = all
18482025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/instances versions = all
18492025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/instances versions = all
18502025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/instances/{instance} versions = all
18512025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/instances/{instance} versions = all
18522025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/instances/{instance} versions = all
18532025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/instances/{instance}/disks versions = all
18542025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/instances/{instance}/disks/attach versions = all
18552025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/instances/{instance}/disks/detach versions = all
18562025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/instances/{instance}/external-ips versions = all
18572025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
18582025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
18592025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/instances/{instance}/reboot versions = all
18602025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/instances/{instance}/serial-console versions = all
18612025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
18622025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
18632025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/instances/{instance}/start versions = all
18642025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/instances/{instance}/stop versions = all
18652025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/internet-gateway-ip-addresses versions = all
18662025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/internet-gateway-ip-addresses versions = all
18672025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
18682025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/internet-gateway-ip-pools versions = all
18692025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/internet-gateway-ip-pools versions = all
18702025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
18712025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/internet-gateways versions = all
18722025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/internet-gateways versions = all
18732025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
18742025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/internet-gateways/{gateway} versions = all
18752025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/ip-pools versions = all
18762025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/ip-pools/{pool} versions = all
18772025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/login/{silo_name}/local versions = all
18782025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/logout versions = all
18792025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/me versions = all
18802025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/me/groups versions = all
18812025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/me/ssh-keys versions = all
18822025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/me/ssh-keys versions = all
18832025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
18842025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
18852025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/metrics/{metric_name} versions = all
18862025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/network-interfaces versions = all
18872025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/network-interfaces versions = all
18882025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/network-interfaces/{interface} versions = all
18892025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/network-interfaces/{interface} versions = all
18902025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/network-interfaces/{interface} versions = all
18912025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/ping versions = all
18922025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/policy versions = all
18932025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/policy versions = all
18942025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/projects versions = all
18952025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/projects versions = all
18962025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/projects/{project} versions = all
18972025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/projects/{project} versions = all
18982025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/projects/{project} versions = all
18992025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/projects/{project}/policy versions = all
19002025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/projects/{project}/policy versions = all
19012025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/snapshots versions = all
19022025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/snapshots versions = all
19032025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/snapshots/{snapshot} versions = all
19042025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/snapshots/{snapshot} versions = all
19052025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/disks versions = all
19062025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
19072025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
19082025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/racks versions = all
19092025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
19102025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/sleds versions = all
19112025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/hardware/sleds versions = all
19122025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
19132025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
19142025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
19152025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
19162025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
19172025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/switch-port versions = all
19182025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
19192025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
19202025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
19212025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
19222025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
19232025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/switches versions = all
19242025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
19252025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/identity-providers versions = all
19262025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/identity-providers/local/users versions = all
19272025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
19282025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
19292025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/identity-providers/saml versions = all
19302025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
19312025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/ip-pools versions = all
19322025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/ip-pools versions = all
19332025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
19342025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/ip-pools/{pool} versions = all
19352025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/system/ip-pools/{pool} versions = all
19362025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
19372025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
19382025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
19392025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
19402025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
19412025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
19422025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
19432025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
19442025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/ip-pools-service versions = all
19452025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/ip-pools-service/ranges versions = all
19462025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
19472025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
19482025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/metrics/{metric_name} versions = all
19492025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/address-lot versions = all
19502025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/networking/address-lot versions = all
19512025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
19522025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
19532025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/allow-list versions = all
19542025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/system/networking/allow-list versions = all
19552025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/networking/bfd-disable versions = all
19562025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/networking/bfd-enable versions = all
19572025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/bfd-status versions = all
19582025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/system/networking/bgp versions = all
19592025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/bgp versions = all
19602025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/networking/bgp versions = all
19612025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/bgp-announce-set versions = all
19622025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
19632025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
19642025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
19652025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/bgp-exported versions = all
19662025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/bgp-message-history versions = all
19672025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
19682025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/bgp-status versions = all
19692025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/loopback-address versions = all
19702025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/networking/loopback-address versions = all
19712025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
19722025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
19732025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/switch-port-settings versions = all
19742025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/networking/switch-port-settings versions = all
19752025-03-21T02:05:27.320ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
19762025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/policy versions = all
19772025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/system/policy versions = all
19782025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/roles versions = all
19792025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/roles/{role_name} versions = all
19802025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/silo-quotas versions = all
19812025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/silos versions = all
19822025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/silos versions = all
19832025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/system/silos/{silo} versions = all
19842025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/silos/{silo} versions = all
19852025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
19862025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/silos/{silo}/policy versions = all
19872025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/system/silos/{silo}/policy versions = all
19882025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/silos/{silo}/quotas versions = all
19892025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
19902025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/system/timeseries/query versions = all
19912025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/timeseries/schemas versions = all
19922025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/system/update/repository versions = all
19932025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/update/repository/{system_version} versions = all
19942025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/update/target-release versions = all
19952025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/system/update/target-release versions = all
19962025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/users versions = all
19972025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/users/{user_id} versions = all
19982025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/users-builtin versions = all
19992025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/users-builtin/{user} versions = all
20002025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/utilization/silos versions = all
20012025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/system/utilization/silos/{silo} versions = all
20022025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/timeseries/query versions = all
20032025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/users versions = all
20042025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/utilization versions = all
20052025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/vpc-firewall-rules versions = all
20062025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/vpc-firewall-rules versions = all
20072025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/vpc-router-routes versions = all
20082025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/vpc-router-routes versions = all
20092025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
20102025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/vpc-router-routes/{route} versions = all
20112025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/vpc-router-routes/{route} versions = all
20122025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/vpc-routers versions = all
20132025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/vpc-routers versions = all
20142025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/vpc-routers/{router} versions = all
20152025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/vpc-routers/{router} versions = all
20162025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/vpc-routers/{router} versions = all
20172025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/vpc-subnets versions = all
20182025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/vpc-subnets versions = all
20192025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
20202025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/vpc-subnets/{subnet} versions = all
20212025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
20222025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
20232025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/vpcs versions = all
20242025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = POST path = /v1/vpcs versions = all
20252025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = DELETE path = /v1/vpcs/{vpc} versions = all
20262025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = GET path = /v1/vpcs/{vpc} versions = all
20272025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.7:443 method = PUT path = /v1/vpcs/{vpc} versions = all
20282025-03-21T02:05:27.321ZINFOnexus (dropshot_external): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 172.30.2.7:443
20292025-03-21T02:05:27.321ZDEBGnexus (dropshot_external): successfully registered DTrace USDT probes local_addr = 172.30.2.7:443
20302025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = / versions = all
20312025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /access versions = all
20322025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /assets/{path} versions = all
20332025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /device/auth versions = all
20342025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /device/confirm versions = all
20352025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /device/success versions = all
20362025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /device/token versions = all
20372025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /device/verify versions = all
20382025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /experimental/v1/probes versions = all
20392025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /experimental/v1/probes versions = all
20402025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /experimental/v1/probes/{probe} versions = all
20412025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /experimental/v1/probes/{probe} versions = all
20422025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /experimental/v1/system/support-bundles versions = all
20432025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /experimental/v1/system/support-bundles versions = all
20442025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
20452025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
20462025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
20472025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
20482025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
20492025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
20502025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
20512025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /images versions = all
20522025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /login versions = all
20532025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /login/{silo_name}/local versions = all
20542025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
20552025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
20562025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
20572025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /lookup/{path} versions = all
20582025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /projects/{path} versions = all
20592025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /projects-new versions = all
20602025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /settings/{path} versions = all
20612025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /system/{path} versions = all
20622025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /utilization versions = all
20632025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/affinity-groups versions = all
20642025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/affinity-groups versions = all
20652025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
20662025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
20672025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
20682025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
20692025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
20702025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
20712025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
20722025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/anti-affinity-groups versions = all
20732025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/anti-affinity-groups versions = all
20742025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
20752025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
20762025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
20772025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
20782025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
20792025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
20802025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
20812025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/certificates versions = all
20822025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/certificates versions = all
20832025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/certificates/{certificate} versions = all
20842025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/certificates/{certificate} versions = all
20852025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/disks versions = all
20862025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/disks versions = all
20872025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/disks/{disk} versions = all
20882025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/disks/{disk} versions = all
20892025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/disks/{disk}/bulk-write versions = all
20902025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
20912025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
20922025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/disks/{disk}/finalize versions = all
20932025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
20942025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/floating-ips versions = all
20952025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/floating-ips versions = all
20962025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
20972025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/floating-ips/{floating_ip} versions = all
20982025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
20992025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
21002025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
21012025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/groups versions = all
21022025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/groups/{group_id} versions = all
21032025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/images versions = all
21042025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/images versions = all
21052025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/images/{image} versions = all
21062025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/images/{image} versions = all
21072025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/images/{image}/demote versions = all
21082025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/images/{image}/promote versions = all
21092025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/instances versions = all
21102025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/instances versions = all
21112025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/instances/{instance} versions = all
21122025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/instances/{instance} versions = all
21132025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/instances/{instance} versions = all
21142025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/instances/{instance}/disks versions = all
21152025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/instances/{instance}/disks/attach versions = all
21162025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/instances/{instance}/disks/detach versions = all
21172025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/instances/{instance}/external-ips versions = all
21182025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
21192025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
21202025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/instances/{instance}/reboot versions = all
21212025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/instances/{instance}/serial-console versions = all
21222025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
21232025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
21242025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/instances/{instance}/start versions = all
21252025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/instances/{instance}/stop versions = all
21262025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/internet-gateway-ip-addresses versions = all
21272025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/internet-gateway-ip-addresses versions = all
21282025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
21292025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/internet-gateway-ip-pools versions = all
21302025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/internet-gateway-ip-pools versions = all
21312025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
21322025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/internet-gateways versions = all
21332025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/internet-gateways versions = all
21342025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
21352025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/internet-gateways/{gateway} versions = all
21362025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/ip-pools versions = all
21372025-03-21T02:05:27.328ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/ip-pools/{pool} versions = all
21382025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/login/{silo_name}/local versions = all
21392025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/logout versions = all
21402025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/me versions = all
21412025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/me/groups versions = all
21422025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/me/ssh-keys versions = all
21432025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/me/ssh-keys versions = all
21442025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
21452025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
21462025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/metrics/{metric_name} versions = all
21472025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/network-interfaces versions = all
21482025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/network-interfaces versions = all
21492025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/network-interfaces/{interface} versions = all
21502025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/network-interfaces/{interface} versions = all
21512025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/network-interfaces/{interface} versions = all
21522025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/ping versions = all
21532025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/policy versions = all
21542025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/policy versions = all
21552025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/projects versions = all
21562025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/projects versions = all
21572025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/projects/{project} versions = all
21582025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/projects/{project} versions = all
21592025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/projects/{project} versions = all
21602025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/projects/{project}/policy versions = all
21612025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/projects/{project}/policy versions = all
21622025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/snapshots versions = all
21632025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/snapshots versions = all
21642025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/snapshots/{snapshot} versions = all
21652025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/snapshots/{snapshot} versions = all
21662025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/disks versions = all
21672025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
21682025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
21692025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/racks versions = all
21702025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
21712025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/sleds versions = all
21722025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/hardware/sleds versions = all
21732025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
21742025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
21752025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
21762025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
21772025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
21782025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/switch-port versions = all
21792025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
21802025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
21812025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
21822025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
21832025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
21842025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/switches versions = all
21852025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
21862025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/identity-providers versions = all
21872025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/identity-providers/local/users versions = all
21882025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
21892025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
21902025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/identity-providers/saml versions = all
21912025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
21922025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/ip-pools versions = all
21932025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/ip-pools versions = all
21942025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
21952025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/ip-pools/{pool} versions = all
21962025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/system/ip-pools/{pool} versions = all
21972025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
21982025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
21992025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
22002025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
22012025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
22022025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
22032025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
22042025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
22052025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/ip-pools-service versions = all
22062025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/ip-pools-service/ranges versions = all
22072025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
22082025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
22092025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/metrics/{metric_name} versions = all
22102025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/address-lot versions = all
22112025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/networking/address-lot versions = all
22122025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
22132025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
22142025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/allow-list versions = all
22152025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/system/networking/allow-list versions = all
22162025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/networking/bfd-disable versions = all
22172025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/networking/bfd-enable versions = all
22182025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/bfd-status versions = all
22192025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/system/networking/bgp versions = all
22202025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/bgp versions = all
22212025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/networking/bgp versions = all
22222025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/bgp-announce-set versions = all
22232025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
22242025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
22252025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
22262025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/bgp-exported versions = all
22272025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/bgp-message-history versions = all
22282025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
22292025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/bgp-status versions = all
22302025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/loopback-address versions = all
22312025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/networking/loopback-address versions = all
22322025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
22332025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
22342025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/switch-port-settings versions = all
22352025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/networking/switch-port-settings versions = all
22362025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
22372025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/policy versions = all
22382025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/system/policy versions = all
22392025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/roles versions = all
22402025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/roles/{role_name} versions = all
22412025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/silo-quotas versions = all
22422025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/silos versions = all
22432025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/silos versions = all
22442025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/system/silos/{silo} versions = all
22452025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/silos/{silo} versions = all
22462025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
22472025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/silos/{silo}/policy versions = all
22482025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/system/silos/{silo}/policy versions = all
22492025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/silos/{silo}/quotas versions = all
22502025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
22512025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/system/timeseries/query versions = all
22522025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/timeseries/schemas versions = all
22532025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/system/update/repository versions = all
22542025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/update/repository/{system_version} versions = all
22552025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/update/target-release versions = all
22562025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/system/update/target-release versions = all
22572025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/users versions = all
22582025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/users/{user_id} versions = all
22592025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/users-builtin versions = all
22602025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/users-builtin/{user} versions = all
22612025-03-21T02:05:27.329ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/utilization/silos versions = all
22622025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/system/utilization/silos/{silo} versions = all
22632025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/timeseries/query versions = all
22642025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/users versions = all
22652025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/utilization versions = all
22662025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/vpc-firewall-rules versions = all
22672025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/vpc-firewall-rules versions = all
22682025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/vpc-router-routes versions = all
22692025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/vpc-router-routes versions = all
22702025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
22712025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/vpc-router-routes/{route} versions = all
22722025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/vpc-router-routes/{route} versions = all
22732025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/vpc-routers versions = all
22742025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/vpc-routers versions = all
22752025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/vpc-routers/{router} versions = all
22762025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/vpc-routers/{router} versions = all
22772025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/vpc-routers/{router} versions = all
22782025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/vpc-subnets versions = all
22792025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/vpc-subnets versions = all
22802025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
22812025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/vpc-subnets/{subnet} versions = all
22822025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
22832025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
22842025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/vpcs versions = all
22852025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = POST path = /v1/vpcs versions = all
22862025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = DELETE path = /v1/vpcs/{vpc} versions = all
22872025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = GET path = /v1/vpcs/{vpc} versions = all
22882025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::c]:12228 method = PUT path = /v1/vpcs/{vpc} versions = all
22892025-03-21T02:05:27.330ZINFOnexus (dropshot_external_techport): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [fd00:1122:3344:101::c]:12228
22902025-03-21T02:05:27.330ZDEBGnexus (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::c]:12228
22912025-03-21T02:05:27.333ZDEBGnexus (nexus-producer-server): registered DTrace probes
22922025-03-21T02:05:27.337ZDEBGnexus (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::c]:42506 method = GET path = /{producer_id} versions = all
22932025-03-21T02:05:27.337ZINFOnexus (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [fd00:1122:3344:101::c]:42506
22942025-03-21T02:05:27.337ZDEBGnexus (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::c]:42506
22952025-03-21T02:05:27.337ZDEBGnexus (nexus-producer-server): Requested any available port, Dropshot server has been bound to [fd00:1122:3344:101::c]:42506
22962025-03-21T02:05:27.337ZDEBGnexus (nexus-producer-server): Nexus IP provided explicitly, will use it addr = [fd00:1122:3344:101::c]:12221
22972025-03-21T02:05:27.337ZDEBGnexus (nexus-producer-server): starting producer registration task
22982025-03-21T02:05:27.337ZINFOnexus (nexus-producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::c]:42506 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = e05eb799-8cf0-4702-bd3b-9994ae86118a
22992025-03-21T02:05:27.337ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
23002025-03-21T02:05:27.337ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
23012025-03-21T02:05:27.463ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
23022025-03-21T02:05:27.464ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "9fb33fe6-5c8a-4f0a-9789-b3583012d437", "date": "Fri, 21 Mar 2025 02:05:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23032025-03-21T02:05:27.464ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23042025-03-21T02:05:27.465ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "4c739241-88e5-4259-9fbf-7564cf096308", "date": "Fri, 21 Mar 2025 02:05:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23052025-03-21T02:05:27.465ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23062025-03-21T02:05:27.469ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23072025-03-21T02:05:27.469ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 266.624532ms iteration = 4 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
23082025-03-21T02:05:27.515ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
23092025-03-21T02:05:27.515ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "8944d90f-d73f-4550-bdba-7f28feaf955b", "content-length": "691", "date": "Fri, 21 Mar 2025 02:05:27 GMT"} })
23102025-03-21T02:05:27.515ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
23112025-03-21T02:05:27.515ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
23122025-03-21T02:05:27.515ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
23132025-03-21T02:05:27.517ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 600.84415ms iteration = 3 status = {}
23142025-03-21T02:05:27.525ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
23152025-03-21T02:05:27.548ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23162025-03-21T02:05:27.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23172025-03-21T02:05:27.551ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
23182025-03-21T02:05:27.551ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23192025-03-21T02:05:27.551ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23202025-03-21T02:05:27.562ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
23212025-03-21T02:05:27.562ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "d00685d6-47c6-4797-b5df-8c636b18a516", "content-length": "134", "date": "Fri, 21 Mar 2025 02:05:27 GMT"} })
23222025-03-21T02:05:27.562ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
23232025-03-21T02:05:27.562ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
23242025-03-21T02:05:27.562ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
23252025-03-21T02:05:27.562ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
23262025-03-21T02:05:27.562ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
23272025-03-21T02:05:27.562ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
23282025-03-21T02:05:27.562ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
23292025-03-21T02:05:27.562ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
23302025-03-21T02:05:27.563ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
23312025-03-21T02:05:27.563ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
23322025-03-21T02:05:27.563ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
23332025-03-21T02:05:27.563ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23342025-03-21T02:05:27.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23352025-03-21T02:05:27.565ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
23362025-03-21T02:05:27.565ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "4897ef7b-9962-4962-ba70-11576c176526", "content-length": "2", "date": "Fri, 21 Mar 2025 02:05:27 GMT"} })
23372025-03-21T02:05:27.565ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
23382025-03-21T02:05:27.565ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf6e6a7c-5235-42e5-8ddc-7988da3c2b7d", "content-length": "52", "date": "Fri, 21 Mar 2025 02:05:27 GMT"} })
23392025-03-21T02:05:27.565ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
23402025-03-21T02:05:27.565ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
23412025-03-21T02:05:27.569ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
23422025-03-21T02:05:27.569ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:35988
23432025-03-21T02:05:27.589ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:35988 req_id = 134cd4ba-0546-4989-862a-3250072bfd3f roles = RoleSet { roles: {} } uri = /metrics/producers
23442025-03-21T02:05:27.589ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:35988 req_id = 134cd4ba-0546-4989-862a-3250072bfd3f resource = Database result = Ok(()) uri = /metrics/producers
23452025-03-21T02:05:27.590ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
23462025-03-21T02:05:27.590ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "186d5daf-6f58-4094-8d35-da9e71b97e0a", "date": "Fri, 21 Mar 2025 02:05:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
23472025-03-21T02:05:27.590ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
23482025-03-21T02:05:27.591ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "d8437a30-f71c-4481-a811-16ce8d92f367", "date": "Fri, 21 Mar 2025 02:05:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
23492025-03-21T02:05:27.591ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
23502025-03-21T02:05:27.592ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "55dafe14-a0e3-46bf-9f1c-1f23953caa68", "date": "Fri, 21 Mar 2025 02:05:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
23512025-03-21T02:05:27.592ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
23522025-03-21T02:05:27.593ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
23532025-03-21T02:05:27.593ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 500.099846ms iteration = 4 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
23542025-03-21T02:05:27.627ZINFOnexus: assigned collector to new producer collector_id = a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53 file = nexus/src/app/oximeter.rs:76 producer_id = e05eb799-8cf0-4702-bd3b-9994ae86118a
23552025-03-21T02:05:27.637ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 64188 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:35988 req_id = 134cd4ba-0546-4989-862a-3250072bfd3f response_code = 201 uri = /metrics/producers
23562025-03-21T02:05:27.637ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "134cd4ba-0546-4989-862a-3250072bfd3f", "content-length": "41", "date": "Fri, 21 Mar 2025 02:05:27 GMT"} })
23572025-03-21T02:05:27.639ZDEBGnexus (producer-registration-task): registered with nexus successfully
23582025-03-21T02:05:27.639ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
23592025-03-21T02:05:27.671ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "436c2e00-bb77-431b-a392-e24b032aecb0", "date": "Fri, 21 Mar 2025 02:05:27 GMT"} })
23602025-03-21T02:05:27.672ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
23612025-03-21T02:05:27.672ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
23622025-03-21T02:05:27.672ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c429048-9d6a-4c9b-ab8d-b8530cd9e6aa", "content-length": "682", "date": "Fri, 21 Mar 2025 02:05:27 GMT"} })
23632025-03-21T02:05:27.672ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23642025-03-21T02:05:27.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23652025-03-21T02:05:27.676ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23662025-03-21T02:05:27.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23672025-03-21T02:05:27.680ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23682025-03-21T02:05:27.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23692025-03-21T02:05:27.682ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23702025-03-21T02:05:27.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23712025-03-21T02:05:27.688ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 772.755193ms iteration = 3 status = {}
23722025-03-21T02:05:36.916ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 8 reason = Timeout
23732025-03-21T02:05:36.916ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23742025-03-21T02:05:36.916ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 8 reason = Timeout
23752025-03-21T02:05:36.916ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23762025-03-21T02:05:36.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23772025-03-21T02:05:36.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23782025-03-21T02:05:36.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23792025-03-21T02:05:36.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23802025-03-21T02:05:36.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23812025-03-21T02:05:36.920ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.227134ms iteration = 8 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
23822025-03-21T02:05:36.921ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
23832025-03-21T02:05:36.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23842025-03-21T02:05:36.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23852025-03-21T02:05:36.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23862025-03-21T02:05:36.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23872025-03-21T02:05:36.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23882025-03-21T02:05:36.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
23892025-03-21T02:05:36.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23902025-03-21T02:05:36.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23912025-03-21T02:05:36.936ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
23922025-03-21T02:05:36.936ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.426806ms iteration = 8 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
23932025-03-21T02:05:46.916ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 9 reason = Timeout
23942025-03-21T02:05:46.916ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23952025-03-21T02:05:46.916ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 9 reason = Timeout
23962025-03-21T02:05:46.916ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23972025-03-21T02:05:46.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23982025-03-21T02:05:46.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23992025-03-21T02:05:46.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24002025-03-21T02:05:46.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24012025-03-21T02:05:46.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24022025-03-21T02:05:46.921ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24032025-03-21T02:05:46.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24042025-03-21T02:05:46.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24052025-03-21T02:05:46.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24062025-03-21T02:05:46.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24072025-03-21T02:05:46.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24082025-03-21T02:05:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
24092025-03-21T02:05:46.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24102025-03-21T02:05:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24112025-03-21T02:05:46.937ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
24122025-03-21T02:05:46.937ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.049856ms iteration = 9 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
24132025-03-21T02:05:46.946ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 30.44034ms iteration = 9 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
24142025-03-21T02:05:49.717ZINFOnexus (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:42506 remote_addr = [fd00:1122:3344:101::d]:64804
24152025-03-21T02:05:49.726ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 9116 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 1ff2037e-4cc9-41fb-bb0f-21cf81b958ee response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
24162025-03-21T02:05:54.707ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:34319
24172025-03-21T02:05:54.719ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:34319 req_id = 43c94ca0-4460-44f4-a30e-9c579cf84fed roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
24182025-03-21T02:05:54.719ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:34319 req_id = 43c94ca0-4460-44f4-a30e-9c579cf84fed resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
24192025-03-21T02:05:54.725ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 18132 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:34319 req_id = 43c94ca0-4460-44f4-a30e-9c579cf84fed response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
24202025-03-21T02:05:54.728ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:34319 req_id = 2fd3a802-6f72-4b5e-949d-f80733cd3ae0 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
24212025-03-21T02:05:54.728ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:34319 req_id = 2fd3a802-6f72-4b5e-949d-f80733cd3ae0 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
24222025-03-21T02:05:54.731ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 5117 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:34319 req_id = 2fd3a802-6f72-4b5e-949d-f80733cd3ae0 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
24232025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 10 reason = Timeout
24242025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24252025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 4 reason = Timeout
24262025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 4 reason = Timeout
24272025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
24282025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 4 reason = Timeout
24292025-03-21T02:05:56.916ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
24302025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 4 reason = Timeout
24312025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
24322025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 5 reason = Timeout
24332025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24342025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
24352025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 10 reason = Timeout
24362025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 4 reason = Timeout
24372025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
24382025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24392025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 4 reason = Timeout
24402025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 4 reason = Timeout
24412025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
24422025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
24432025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 4 reason = Timeout
24442025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
24452025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 4 reason = Timeout
24462025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24472025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 4 reason = Timeout
24482025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
24492025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
24502025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 4 reason = Timeout
24512025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
24522025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 4 reason = Timeout
24532025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24542025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 4 reason = Timeout
24552025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 4 reason = Timeout
24562025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24572025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
24582025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 4 reason = Timeout
24592025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
24602025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 4 reason = Timeout
24612025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
24622025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24632025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
24642025-03-21T02:05:56.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
24652025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
24662025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
24672025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24682025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24692025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24702025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
24712025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
24722025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
24732025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
24742025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
24752025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24762025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
24772025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24782025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
24792025-03-21T02:05:56.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
24802025-03-21T02:05:56.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24812025-03-21T02:05:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24822025-03-21T02:05:56.920ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24832025-03-21T02:05:56.921ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.112399ms iteration = 4 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
24842025-03-21T02:05:56.921ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1065, tv_nsec: 547069532 } })
24852025-03-21T02:05:56.921ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
24862025-03-21T02:05:56.922ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24872025-03-21T02:05:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
24882025-03-21T02:05:56.922ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
24892025-03-21T02:05:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
24902025-03-21T02:05:56.924ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
24912025-03-21T02:05:56.924ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24922025-03-21T02:05:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24932025-03-21T02:05:56.925ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1065, tv_nsec: 551489969 } })
24942025-03-21T02:05:56.926ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 9.663649ms iteration = 4 status = {"errors":[],"garbage_collect_requested":[]}
24952025-03-21T02:05:56.926ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
24962025-03-21T02:05:56.927ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 10.669586ms iteration = 4 status = {"errors":[],"finish_invoked_ok":[]}
24972025-03-21T02:05:56.927ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24982025-03-21T02:05:56.929ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24992025-03-21T02:05:56.929ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25002025-03-21T02:05:56.930ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 13.968083ms iteration = 10 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25012025-03-21T02:05:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
25022025-03-21T02:05:56.930ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25032025-03-21T02:05:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25042025-03-21T02:05:56.932ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
25052025-03-21T02:05:56.932ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
25062025-03-21T02:05:56.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
25072025-03-21T02:05:56.932ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
25082025-03-21T02:05:56.932ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
25092025-03-21T02:05:56.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
25102025-03-21T02:05:56.934ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
25112025-03-21T02:05:56.934ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25122025-03-21T02:05:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25132025-03-21T02:05:56.935ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1065, tv_nsec: 561449132 } })
25142025-03-21T02:05:56.936ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
25152025-03-21T02:05:56.936ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
25162025-03-21T02:05:56.936ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25172025-03-21T02:05:56.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25182025-03-21T02:05:56.939ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25192025-03-21T02:05:56.939ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
25202025-03-21T02:05:56.939ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 23.253795ms iteration = 4 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
25212025-03-21T02:05:56.940ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
25222025-03-21T02:05:56.940ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
25232025-03-21T02:05:56.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25242025-03-21T02:05:56.964ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
25252025-03-21T02:05:56.964ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
25262025-03-21T02:05:56.964ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
25272025-03-21T02:05:56.965ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
25282025-03-21T02:05:56.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
25292025-03-21T02:05:56.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
25302025-03-21T02:05:56.966ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 49.735446ms iteration = 4 status = {"errors":[],"requests_created_ok":[]}
25312025-03-21T02:05:56.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25322025-03-21T02:05:56.968ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
25332025-03-21T02:05:56.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
25342025-03-21T02:05:56.969ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
25352025-03-21T02:05:56.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25362025-03-21T02:05:56.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25372025-03-21T02:05:56.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
25382025-03-21T02:05:56.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25392025-03-21T02:05:56.972ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 55.694139ms iteration = 4 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
25402025-03-21T02:05:56.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25412025-03-21T02:05:56.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25422025-03-21T02:05:56.974ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25432025-03-21T02:05:56.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25442025-03-21T02:05:56.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
25452025-03-21T02:05:56.976ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25462025-03-21T02:05:56.977ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 60.689227ms iteration = 4 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
25472025-03-21T02:05:56.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25482025-03-21T02:05:56.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
25492025-03-21T02:05:56.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25502025-03-21T02:05:56.978ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25512025-03-21T02:05:56.978ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
25522025-03-21T02:05:56.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25532025-03-21T02:05:56.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25542025-03-21T02:05:56.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
25552025-03-21T02:05:56.980ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25562025-03-21T02:05:56.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
25572025-03-21T02:05:56.982ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
25582025-03-21T02:05:56.982ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
25592025-03-21T02:05:56.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
25602025-03-21T02:05:56.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
25612025-03-21T02:05:56.989ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
25622025-03-21T02:05:56.989ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 72.96849ms iteration = 10 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
25632025-03-21T02:05:56.989ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 73.307297ms iteration = 5 status = {"physical_disks_added":0}
25642025-03-21T02:05:56.996ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25652025-03-21T02:05:56.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
25662025-03-21T02:05:57.004ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
25672025-03-21T02:05:57.004ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
25682025-03-21T02:05:57.004ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
25692025-03-21T02:05:57.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
25702025-03-21T02:05:57.018ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
25712025-03-21T02:05:57.018ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
25722025-03-21T02:05:57.018ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e47f5c34-e8dc-435a-a31d-8d6ef1ac9763", "content-length": "26", "date": "Fri, 21 Mar 2025 02:05:57 GMT"} })
25732025-03-21T02:05:57.018ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
25742025-03-21T02:05:57.019ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
25752025-03-21T02:05:57.026ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 110.160838ms iteration = 4 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
25762025-03-21T02:05:57.038ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
25772025-03-21T02:05:57.038ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 121.492086ms iteration = 4 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
25782025-03-21T02:05:57.041ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
25792025-03-21T02:05:57.041ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
25802025-03-21T02:05:57.041ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 124.825523ms iteration = 4 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
25812025-03-21T02:05:57.107ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25822025-03-21T02:05:57.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
25832025-03-21T02:05:57.116ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
25842025-03-21T02:05:57.116ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
25852025-03-21T02:05:57.117ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d90e44b1-597c-4cc0-9d29-97fc5af345d9", "content-length": "26", "date": "Fri, 21 Mar 2025 02:05:57 GMT"} })
25862025-03-21T02:05:57.117ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
25872025-03-21T02:05:57.118ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
25882025-03-21T02:05:57.119ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
25892025-03-21T02:05:57.119ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
25902025-03-21T02:05:57.119ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "66cb960a-092d-42e0-8f6a-dbbfb27eb800", "content-length": "26", "date": "Fri, 21 Mar 2025 02:05:57 GMT"} })
25912025-03-21T02:05:57.119ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
25922025-03-21T02:05:57.120ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 204.145304ms iteration = 4 status = {}
25932025-03-21T02:05:57.120ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
25942025-03-21T02:05:57.127ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
25952025-03-21T02:05:57.127ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
25962025-03-21T02:05:57.128ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8a2294b-84dc-42da-88d6-15707ca2a0b9", "content-length": "372", "date": "Fri, 21 Mar 2025 02:05:57 GMT"} })
25972025-03-21T02:05:57.128ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25982025-03-21T02:05:57.128ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25992025-03-21T02:05:57.143ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26002025-03-21T02:05:57.145ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
26012025-03-21T02:05:57.145ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26022025-03-21T02:05:57.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26032025-03-21T02:05:57.148ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
26042025-03-21T02:05:57.148ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
26052025-03-21T02:05:57.149ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "453a9f52-92aa-424b-9eac-0bd6926a3593", "date": "Fri, 21 Mar 2025 02:05:57 GMT"} })
26062025-03-21T02:05:57.149ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26072025-03-21T02:05:57.149ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26082025-03-21T02:05:57.151ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26092025-03-21T02:05:57.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26102025-03-21T02:05:57.169ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
26112025-03-21T02:05:57.169ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "28020338-0efb-40fb-aa9f-6047b0601cf0", "content-length": "1", "date": "Fri, 21 Mar 2025 02:05:56 GMT"} })
26122025-03-21T02:05:57.170ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
26132025-03-21T02:05:57.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
26142025-03-21T02:05:57.172ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 255.800063ms iteration = 4 status = null
26152025-03-21T02:05:57.183ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26162025-03-21T02:05:57.185ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
26172025-03-21T02:05:57.185ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26182025-03-21T02:05:57.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26192025-03-21T02:05:57.248ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26202025-03-21T02:05:57.248ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
26212025-03-21T02:05:57.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26222025-03-21T02:05:57.249ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "5871167e-d5c0-48c5-ab5c-7ccacde85819", "content-length": "691", "date": "Fri, 21 Mar 2025 02:05:57 GMT"} })
26232025-03-21T02:05:57.249ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
26242025-03-21T02:05:57.249ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
26252025-03-21T02:05:57.249ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
26262025-03-21T02:05:57.250ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 334.34631ms iteration = 4 status = {}
26272025-03-21T02:05:57.252ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26282025-03-21T02:05:57.253ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
26292025-03-21T02:05:57.253ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
26302025-03-21T02:05:57.253ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26312025-03-21T02:05:57.253ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "7fa40f8a-6240-467a-b715-d4a7260ae8ff", "date": "Fri, 21 Mar 2025 02:05:57 GMT"} })
26322025-03-21T02:05:57.255ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 338.649515ms iteration = 4 status = {}
26332025-03-21T02:05:57.255ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
26342025-03-21T02:05:57.255ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26352025-03-21T02:05:57.255ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26362025-03-21T02:05:57.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26372025-03-21T02:05:57.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26382025-03-21T02:05:57.257ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26392025-03-21T02:05:57.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26402025-03-21T02:05:57.259ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26412025-03-21T02:05:57.261ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
26422025-03-21T02:05:57.261ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26432025-03-21T02:05:57.261ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26442025-03-21T02:05:57.263ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 347.062232ms iteration = 4 status = 0
26452025-03-21T02:05:57.265ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
26462025-03-21T02:05:57.266ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7f9cae1-5a93-4937-9b94-9a9c2f7f3478", "content-length": "134", "date": "Fri, 21 Mar 2025 02:05:56 GMT"} })
26472025-03-21T02:05:57.266ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
26482025-03-21T02:05:57.266ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
26492025-03-21T02:05:57.266ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
26502025-03-21T02:05:57.266ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
26512025-03-21T02:05:57.266ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
26522025-03-21T02:05:57.266ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
26532025-03-21T02:05:57.266ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
26542025-03-21T02:05:57.266ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
26552025-03-21T02:05:57.266ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
26562025-03-21T02:05:57.266ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
26572025-03-21T02:05:57.266ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
26582025-03-21T02:05:57.266ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26592025-03-21T02:05:57.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26602025-03-21T02:05:57.269ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
26612025-03-21T02:05:57.269ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0bd9825-a0bb-4825-9f9c-4aabd6b75b3f", "content-length": "2", "date": "Fri, 21 Mar 2025 02:05:56 GMT"} })
26622025-03-21T02:05:57.269ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
26632025-03-21T02:05:57.269ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffbd16a6-a3e5-46b6-a2ff-bfa00c8a1bb7", "content-length": "52", "date": "Fri, 21 Mar 2025 02:05:56 GMT"} })
26642025-03-21T02:05:57.269ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
26652025-03-21T02:05:57.269ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
26662025-03-21T02:05:57.368ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d32a3347-68fc-4788-8dd3-627f19a8c694", "date": "Fri, 21 Mar 2025 02:05:57 GMT"} })
26672025-03-21T02:05:57.368ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
26682025-03-21T02:05:57.368ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
26692025-03-21T02:05:57.369ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f982ec25-3398-4da1-ae8d-869a4c06f815", "content-length": "682", "date": "Fri, 21 Mar 2025 02:05:57 GMT"} })
26702025-03-21T02:05:57.369ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26712025-03-21T02:05:57.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26722025-03-21T02:05:57.372ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26732025-03-21T02:05:57.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26742025-03-21T02:05:57.377ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26752025-03-21T02:05:57.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26762025-03-21T02:05:57.379ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26772025-03-21T02:05:57.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26782025-03-21T02:05:57.386ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 470.049623ms iteration = 4 status = {}
26792025-03-21T02:05:59.716ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 255 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 0a034741-d5e3-404a-94c3-2e4e984996de response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
26802025-03-21T02:06:06.917ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 11 reason = Timeout
26812025-03-21T02:06:06.917ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 11 reason = Timeout
26822025-03-21T02:06:06.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26832025-03-21T02:06:06.917ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26842025-03-21T02:06:06.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26852025-03-21T02:06:06.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26862025-03-21T02:06:06.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26872025-03-21T02:06:06.921ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
26882025-03-21T02:06:06.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26892025-03-21T02:06:06.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26902025-03-21T02:06:06.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26912025-03-21T02:06:06.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26922025-03-21T02:06:06.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26932025-03-21T02:06:06.926ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
26942025-03-21T02:06:06.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26952025-03-21T02:06:06.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26962025-03-21T02:06:06.947ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26972025-03-21T02:06:06.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26982025-03-21T02:06:06.948ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.820204ms iteration = 11 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
26992025-03-21T02:06:07.039ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
27002025-03-21T02:06:07.039ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 122.049855ms iteration = 11 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
27012025-03-21T02:06:09.717ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 310 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 469bcf67-d549-4b49-9a24-099a6ba16df7 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
27022025-03-21T02:06:16.916ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 12 reason = Timeout
27032025-03-21T02:06:16.917ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 12 reason = Timeout
27042025-03-21T02:06:16.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27052025-03-21T02:06:16.917ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27062025-03-21T02:06:16.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27072025-03-21T02:06:16.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27082025-03-21T02:06:16.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27092025-03-21T02:06:16.921ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
27102025-03-21T02:06:16.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27112025-03-21T02:06:16.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27122025-03-21T02:06:16.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27132025-03-21T02:06:16.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27142025-03-21T02:06:16.947ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27152025-03-21T02:06:16.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27162025-03-21T02:06:16.949ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.421485ms iteration = 12 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
27172025-03-21T02:06:16.953ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27182025-03-21T02:06:16.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
27192025-03-21T02:06:16.955ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27202025-03-21T02:06:16.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27212025-03-21T02:06:16.964ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
27222025-03-21T02:06:16.964ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 47.889444ms iteration = 12 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
27232025-03-21T02:06:19.717ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 285 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = b2c4c0ac-b009-4dd6-b0f9-945a8b147b11 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
27242025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 13 reason = Timeout
27252025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 5 reason = Timeout
27262025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27272025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
27282025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 4 reason = Timeout
27292025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 3 reason = Timeout
27302025-03-21T02:06:26.916ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
27312025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
27322025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 3 reason = Timeout
27332025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 22.17\u{b5}s iteration = 4 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
27342025-03-21T02:06:26.916ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 01:56:26.916736178 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
27352025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
27362025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 5 reason = Timeout
27372025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
27382025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 5 reason = Timeout
27392025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 13 reason = Timeout
27402025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
27412025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 5 reason = Timeout
27422025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27432025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27442025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 5 reason = Timeout
27452025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 5 reason = Timeout
27462025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
27472025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
27482025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 3 reason = Timeout
27492025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
27502025-03-21T02:06:26.916ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 5 reason = Timeout
27512025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 3 reason = Timeout
27522025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
27532025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27542025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 3 reason = Timeout
27552025-03-21T02:06:26.917ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
27562025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
27572025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 5 reason = Timeout
27582025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
27592025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 3 reason = Timeout
27602025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
27612025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 5 reason = Timeout
27622025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
27632025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 5 reason = Timeout
27642025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
27652025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27662025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 6 reason = Timeout
27672025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27682025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 5 reason = Timeout
27692025-03-21T02:06:26.917ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
27702025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27712025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 5 reason = Timeout
27722025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27732025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 5 reason = Timeout
27742025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27752025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 5 reason = Timeout
27762025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27772025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 5 reason = Timeout
27782025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
27792025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 5 reason = Timeout
27802025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 3 reason = Timeout
27812025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 3 reason = Timeout
27822025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
27832025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
27842025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 3 reason = Timeout
27852025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 3 reason = Timeout
27862025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
27872025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27882025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
27892025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
27902025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
27912025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 5 reason = Timeout
27922025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 5 reason = Timeout
27932025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
27942025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27952025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27962025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
27972025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
27982025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
27992025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
28002025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
28012025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28022025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
28032025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
28042025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
28052025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
28062025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
28072025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
28082025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28092025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
28102025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
28112025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28122025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28132025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
28142025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28152025-03-21T02:06:26.917ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1095, tv_nsec: 543091935 } })
28162025-03-21T02:06:26.917ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1095, tv_nsec: 543092295 } })
28172025-03-21T02:06:26.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
28182025-03-21T02:06:26.918ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
28192025-03-21T02:06:26.918ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.399204ms iteration = 3 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
28202025-03-21T02:06:26.918ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
28212025-03-21T02:06:26.918ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.419915ms iteration = 3 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
28222025-03-21T02:06:26.919ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 01:56:26.916736178 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
28232025-03-21T02:06:26.919ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.150835ms iteration = 3 status = {"expiration":"2025-03-21T01:56:26.916736178Z","pruned":[]}
28242025-03-21T02:06:26.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
28252025-03-21T02:06:26.921ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.912017ms iteration = 5 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
28262025-03-21T02:06:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
28272025-03-21T02:06:26.921ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1095, tv_nsec: 546539965 } })
28282025-03-21T02:06:26.921ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
28292025-03-21T02:06:26.922ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28302025-03-21T02:06:26.922ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
28312025-03-21T02:06:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
28322025-03-21T02:06:26.922ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28332025-03-21T02:06:26.924ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 7.417179ms iteration = 5 status = {"errors":[],"finish_invoked_ok":[]}
28342025-03-21T02:06:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
28352025-03-21T02:06:26.925ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
28362025-03-21T02:06:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
28372025-03-21T02:06:26.925ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28382025-03-21T02:06:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
28392025-03-21T02:06:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28402025-03-21T02:06:26.928ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28412025-03-21T02:06:26.929ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1095, tv_nsec: 554360321 } })
28422025-03-21T02:06:26.929ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
28432025-03-21T02:06:26.929ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 12.853743ms iteration = 5 status = {"errors":[],"garbage_collect_requested":[]}
28442025-03-21T02:06:26.930ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 13.916242ms iteration = 3 status = {"errors":[],"found_port_ok":[]}
28452025-03-21T02:06:26.931ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
28462025-03-21T02:06:26.931ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28472025-03-21T02:06:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28482025-03-21T02:06:26.932ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
28492025-03-21T02:06:26.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
28502025-03-21T02:06:26.935ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28512025-03-21T02:06:26.935ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28522025-03-21T02:06:26.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28532025-03-21T02:06:26.937ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28542025-03-21T02:06:26.938ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
28552025-03-21T02:06:26.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28562025-03-21T02:06:26.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28572025-03-21T02:06:26.940ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
28582025-03-21T02:06:26.940ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
28592025-03-21T02:06:26.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
28602025-03-21T02:06:26.944ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28612025-03-21T02:06:26.944ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28622025-03-21T02:06:26.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28632025-03-21T02:06:26.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
28642025-03-21T02:06:26.948ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28652025-03-21T02:06:26.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28662025-03-21T02:06:26.952ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1095, tv_nsec: 577571525 } })
28672025-03-21T02:06:26.952ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
28682025-03-21T02:06:26.955ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28692025-03-21T02:06:26.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28702025-03-21T02:06:26.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28712025-03-21T02:06:26.977ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
28722025-03-21T02:06:26.977ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28732025-03-21T02:06:26.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28742025-03-21T02:06:26.978ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 61.670834ms iteration = 5 status = {"errors":[],"requests_created_ok":[]}
28752025-03-21T02:06:26.984ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
28762025-03-21T02:06:26.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
28772025-03-21T02:06:26.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28782025-03-21T02:06:26.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28792025-03-21T02:06:26.986ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
28802025-03-21T02:06:26.986ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
28812025-03-21T02:06:26.986ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
28822025-03-21T02:06:26.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
28832025-03-21T02:06:26.992ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28842025-03-21T02:06:26.992ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28852025-03-21T02:06:26.995ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
28862025-03-21T02:06:26.995ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28872025-03-21T02:06:26.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28882025-03-21T02:06:26.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
28892025-03-21T02:06:26.995ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
28902025-03-21T02:06:26.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
28912025-03-21T02:06:26.997ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28922025-03-21T02:06:26.997ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28932025-03-21T02:06:26.999ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
28942025-03-21T02:06:26.999ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
28952025-03-21T02:06:26.999ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 82.488937ms iteration = 3 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
28962025-03-21T02:06:26.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
28972025-03-21T02:06:27.000ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
28982025-03-21T02:06:27.000ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
28992025-03-21T02:06:27.000ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29002025-03-21T02:06:27.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
29012025-03-21T02:06:27.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29022025-03-21T02:06:27.003ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 86.437295ms iteration = 13 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
29032025-03-21T02:06:27.003ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
29042025-03-21T02:06:27.003ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
29052025-03-21T02:06:27.003ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
29062025-03-21T02:06:27.003ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
29072025-03-21T02:06:27.003ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
29082025-03-21T02:06:27.003ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
29092025-03-21T02:06:27.003ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29102025-03-21T02:06:27.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
29112025-03-21T02:06:27.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
29122025-03-21T02:06:27.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
29132025-03-21T02:06:27.004ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 87.32479ms iteration = 5 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
29142025-03-21T02:06:27.007ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
29152025-03-21T02:06:27.007ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
29162025-03-21T02:06:27.007ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
29172025-03-21T02:06:27.007ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 90.73799ms iteration = 3 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
29182025-03-21T02:06:27.009ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
29192025-03-21T02:06:27.009ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
29202025-03-21T02:06:27.009ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
29212025-03-21T02:06:27.009ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
29222025-03-21T02:06:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
29232025-03-21T02:06:27.014ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29242025-03-21T02:06:27.017ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
29252025-03-21T02:06:27.017ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
29262025-03-21T02:06:27.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
29272025-03-21T02:06:27.026ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
29282025-03-21T02:06:27.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
29292025-03-21T02:06:27.036ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
29302025-03-21T02:06:27.036ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
29312025-03-21T02:06:27.036ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
29322025-03-21T02:06:27.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
29332025-03-21T02:06:27.037ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 120.203512ms iteration = 5 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
29342025-03-21T02:06:27.043ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
29352025-03-21T02:06:27.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
29362025-03-21T02:06:27.044ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29372025-03-21T02:06:27.046ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
29382025-03-21T02:06:27.046ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29392025-03-21T02:06:27.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29402025-03-21T02:06:27.068ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
29412025-03-21T02:06:27.068ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 150.900007ms iteration = 5 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
29422025-03-21T02:06:27.095ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
29432025-03-21T02:06:27.097ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
29442025-03-21T02:06:27.097ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
29452025-03-21T02:06:27.097ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "20494295-5b33-4d99-bc4b-53fd9bc8cc72", "content-length": "26", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} })
29462025-03-21T02:06:27.097ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
29472025-03-21T02:06:27.099ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
29482025-03-21T02:06:27.104ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
29492025-03-21T02:06:27.104ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
29502025-03-21T02:06:27.104ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "811e6825-55db-44a8-823a-989cb8d4d271", "content-length": "26", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} })
29512025-03-21T02:06:27.104ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
29522025-03-21T02:06:27.106ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
29532025-03-21T02:06:27.112ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29542025-03-21T02:06:27.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29552025-03-21T02:06:27.129ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
29562025-03-21T02:06:27.129ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
29572025-03-21T02:06:27.129ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "18ea22ba-f2df-4003-86dc-7422721f4ec7", "content-length": "26", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} })
29582025-03-21T02:06:27.129ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
29592025-03-21T02:06:27.130ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
29602025-03-21T02:06:27.138ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
29612025-03-21T02:06:27.203ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29622025-03-21T02:06:27.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29632025-03-21T02:06:27.254ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 337.436994ms iteration = 5 status = {}
29642025-03-21T02:06:27.260ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
29652025-03-21T02:06:27.260ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "a7ce9a37-002c-4eb7-9233-5541fe56697f", "date": "Fri, 21 Mar 2025 02:06:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29662025-03-21T02:06:27.260ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29672025-03-21T02:06:27.261ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "60903500-e647-47ec-a79b-6aad04bffef0", "date": "Fri, 21 Mar 2025 02:06:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29682025-03-21T02:06:27.261ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29692025-03-21T02:06:27.262ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29702025-03-21T02:06:27.262ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 345.523435ms iteration = 5 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
29712025-03-21T02:06:27.278ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
29722025-03-21T02:06:27.279ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "f72d76a4-b4a1-498a-8d2b-cc751198cf73", "content-length": "1", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} })
29732025-03-21T02:06:27.280ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
29742025-03-21T02:06:27.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
29752025-03-21T02:06:27.303ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
29762025-03-21T02:06:27.303ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
29772025-03-21T02:06:27.305ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29782025-03-21T02:06:27.306ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29792025-03-21T02:06:27.307ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
29802025-03-21T02:06:27.307ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
29812025-03-21T02:06:27.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
29822025-03-21T02:06:27.309ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
29832025-03-21T02:06:27.309ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
29842025-03-21T02:06:27.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
29852025-03-21T02:06:27.312ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29862025-03-21T02:06:27.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29872025-03-21T02:06:27.313ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 396.33138ms iteration = 5 status = null
29882025-03-21T02:06:27.314ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29892025-03-21T02:06:27.317ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
29902025-03-21T02:06:27.317ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29912025-03-21T02:06:27.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29922025-03-21T02:06:27.320ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 403.750019ms iteration = 5 status = 0
29932025-03-21T02:06:27.321ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
29942025-03-21T02:06:27.321ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 404.98656ms iteration = 13 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
29952025-03-21T02:06:27.331ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
29962025-03-21T02:06:27.339ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
29972025-03-21T02:06:27.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
29982025-03-21T02:06:27.340ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
29992025-03-21T02:06:27.341ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
30002025-03-21T02:06:27.344ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30012025-03-21T02:06:27.346ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
30022025-03-21T02:06:27.346ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
30032025-03-21T02:06:27.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
30042025-03-21T02:06:27.347ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30052025-03-21T02:06:27.347ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
30062025-03-21T02:06:27.347ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
30072025-03-21T02:06:27.347ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 430.736138ms iteration = 5 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
30082025-03-21T02:06:27.349ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
30092025-03-21T02:06:27.349ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
30102025-03-21T02:06:27.349ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
30112025-03-21T02:06:27.352ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
30122025-03-21T02:06:27.352ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
30132025-03-21T02:06:27.352ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
30142025-03-21T02:06:27.352ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30152025-03-21T02:06:27.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
30162025-03-21T02:06:27.354ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
30172025-03-21T02:06:27.354ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
30182025-03-21T02:06:27.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
30192025-03-21T02:06:27.359ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30202025-03-21T02:06:27.359ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
30212025-03-21T02:06:27.360ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
30222025-03-21T02:06:27.360ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
30232025-03-21T02:06:27.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
30242025-03-21T02:06:27.361ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30252025-03-21T02:06:27.362ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
30262025-03-21T02:06:27.362ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
30272025-03-21T02:06:27.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
30282025-03-21T02:06:27.364ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
30292025-03-21T02:06:27.364ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
30302025-03-21T02:06:27.364ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-21 02:04:31.662541 UTC dns_group = external generation = 2
30312025-03-21T02:06:27.364ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 447.334167ms iteration = 3 status = {"generation":2}
30322025-03-21T02:06:27.364ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
30332025-03-21T02:06:27.364ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
30342025-03-21T02:06:27.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
30352025-03-21T02:06:27.368ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
30362025-03-21T02:06:27.368ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
30372025-03-21T02:06:27.368ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-21 02:04:30.886360 UTC dns_group = internal generation = 1
30382025-03-21T02:06:27.368ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 451.282326ms iteration = 3 status = {"generation":1}
30392025-03-21T02:06:27.382ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 465.068196ms iteration = 5 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
30402025-03-21T02:06:27.386ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30412025-03-21T02:06:27.387ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
30422025-03-21T02:06:27.387ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
30432025-03-21T02:06:27.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
30442025-03-21T02:06:27.388ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
30452025-03-21T02:06:27.388ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 471.534589ms iteration = 3 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
30462025-03-21T02:06:27.392ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
30472025-03-21T02:06:27.392ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 475.283404ms iteration = 5 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
30482025-03-21T02:06:27.398ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30492025-03-21T02:06:27.399ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
30502025-03-21T02:06:27.399ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
30512025-03-21T02:06:27.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
30522025-03-21T02:06:27.406ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
30532025-03-21T02:06:27.408ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "bf1ad03f-35e5-4701-8a37-3f1fff7f6e6c", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
30542025-03-21T02:06:27.408ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
30552025-03-21T02:06:27.409ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "3e1db19a-2c9b-457f-940c-8381bb3799b5", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
30562025-03-21T02:06:27.409ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
30572025-03-21T02:06:27.411ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "ff378c8c-b349-4334-9778-29fe49ee898d", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
30582025-03-21T02:06:27.412ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
30592025-03-21T02:06:27.413ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
30602025-03-21T02:06:27.413ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 496.394491ms iteration = 5 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
30612025-03-21T02:06:27.413ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8fd8e01-9d3e-4187-9c3e-1da600db3a15", "content-length": "372", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} })
30622025-03-21T02:06:27.413ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30632025-03-21T02:06:27.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30642025-03-21T02:06:27.426ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30652025-03-21T02:06:27.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30662025-03-21T02:06:27.428ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 511.203339ms iteration = 6 status = {"physical_disks_added":0}
30672025-03-21T02:06:27.428ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
30682025-03-21T02:06:27.428ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30692025-03-21T02:06:27.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30702025-03-21T02:06:27.439ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
30712025-03-21T02:06:27.439ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a7d6452-791a-47a3-8a7f-30cc5c1a6170", "content-length": "134", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} })
30722025-03-21T02:06:27.439ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
30732025-03-21T02:06:27.439ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
30742025-03-21T02:06:27.439ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) }
30752025-03-21T02:06:27.439ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 522.70842ms iteration = 3 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
30762025-03-21T02:06:27.439ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
30772025-03-21T02:06:27.439ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
30782025-03-21T02:06:27.439ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
30792025-03-21T02:06:27.439ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
30802025-03-21T02:06:27.439ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
30812025-03-21T02:06:27.439ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
30822025-03-21T02:06:27.440ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
30832025-03-21T02:06:27.440ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
30842025-03-21T02:06:27.440ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
30852025-03-21T02:06:27.440ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30862025-03-21T02:06:27.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30872025-03-21T02:06:27.443ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
30882025-03-21T02:06:27.443ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bd7740d-6bb7-40a7-8c75-538ee51ffa2e", "content-length": "2", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} })
30892025-03-21T02:06:27.443ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
30902025-03-21T02:06:27.444ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c4c28ac-fa89-4ad8-9f03-1e739e8cdceb", "content-length": "52", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} })
30912025-03-21T02:06:27.444ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
30922025-03-21T02:06:27.444ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
30932025-03-21T02:06:27.469ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
30942025-03-21T02:06:27.469ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
30952025-03-21T02:06:27.472ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "81b9a6ef-5e3c-49a0-a49f-b9060a23ff38", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} })
30962025-03-21T02:06:27.472ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30972025-03-21T02:06:27.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30982025-03-21T02:06:27.475ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30992025-03-21T02:06:27.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31002025-03-21T02:06:27.478ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31012025-03-21T02:06:27.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31022025-03-21T02:06:27.480ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
31032025-03-21T02:06:27.480ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
31042025-03-21T02:06:27.483ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "49ab5ac9-6314-4cfd-972f-045cfc8783ff", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} })
31052025-03-21T02:06:27.485ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 568.363015ms iteration = 5 status = {}
31062025-03-21T02:06:27.498ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
31072025-03-21T02:06:27.499ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "db4fb8b9-9eb0-47e5-95f6-9b4761e974df", "content-length": "691", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} })
31082025-03-21T02:06:27.499ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
31092025-03-21T02:06:27.499ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
31102025-03-21T02:06:27.499ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
31112025-03-21T02:06:27.500ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 583.347805ms iteration = 5 status = {}
31122025-03-21T02:06:27.604ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "0f3a01bb-3b51-4be0-9303-24c542e2afe2", "date": "Fri, 21 Mar 2025 02:06:27 GMT"} })
31132025-03-21T02:06:27.604ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
31142025-03-21T02:06:27.604ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
31152025-03-21T02:06:27.604ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f24ef6d-d773-46c9-88fc-fa943777f2e6", "content-length": "682", "date": "Fri, 21 Mar 2025 02:06:26 GMT"} })
31162025-03-21T02:06:27.604ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31172025-03-21T02:06:27.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31182025-03-21T02:06:27.608ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31192025-03-21T02:06:27.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31202025-03-21T02:06:27.610ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31212025-03-21T02:06:27.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31222025-03-21T02:06:27.613ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31232025-03-21T02:06:27.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31242025-03-21T02:06:27.619ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 702.334057ms iteration = 5 status = {}
31252025-03-21T02:06:29.717ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 304 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 36dfce80-d7a3-4b43-a3cf-35a247832db4 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
31262025-03-21T02:06:36.916ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 14 reason = Timeout
31272025-03-21T02:06:36.916ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 14 reason = Timeout
31282025-03-21T02:06:36.916ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31292025-03-21T02:06:36.916ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31302025-03-21T02:06:36.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31312025-03-21T02:06:36.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31322025-03-21T02:06:36.945ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31332025-03-21T02:06:36.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31342025-03-21T02:06:36.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31352025-03-21T02:06:36.948ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.231314ms iteration = 14 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
31362025-03-21T02:06:36.949ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
31372025-03-21T02:06:36.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31382025-03-21T02:06:36.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31392025-03-21T02:06:36.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31402025-03-21T02:06:36.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31412025-03-21T02:06:36.953ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31422025-03-21T02:06:36.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
31432025-03-21T02:06:36.955ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31442025-03-21T02:06:36.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31452025-03-21T02:06:37.074ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
31462025-03-21T02:06:37.074ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 157.594104ms iteration = 14 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
31472025-03-21T02:06:39.717ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 281 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = e8b707c9-4499-4962-9238-82544a58ec4f response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
31482025-03-21T02:06:46.916ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 15 reason = Timeout
31492025-03-21T02:06:46.916ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31502025-03-21T02:06:46.916ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 15 reason = Timeout
31512025-03-21T02:06:46.916ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31522025-03-21T02:06:46.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31532025-03-21T02:06:46.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31542025-03-21T02:06:46.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31552025-03-21T02:06:46.921ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
31562025-03-21T02:06:46.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31572025-03-21T02:06:46.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31582025-03-21T02:06:46.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31592025-03-21T02:06:46.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31602025-03-21T02:06:46.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31612025-03-21T02:06:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
31622025-03-21T02:06:46.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31632025-03-21T02:06:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31642025-03-21T02:06:46.934ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
31652025-03-21T02:06:46.934ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.615607ms iteration = 15 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
31662025-03-21T02:06:46.947ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31672025-03-21T02:06:46.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31682025-03-21T02:06:46.982ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 65.620562ms iteration = 15 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
31692025-03-21T02:06:49.717ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 296 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = ba8ec24a-b9da-48c9-bad8-eb0e56b24e03 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
31702025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 16 reason = Timeout
31712025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31722025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 6 reason = Timeout
31732025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 6 reason = Timeout
31742025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 6 reason = Timeout
31752025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
31762025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
31772025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 16 reason = Timeout
31782025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 6 reason = Timeout
31792025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31802025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 6 reason = Timeout
31812025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31822025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
31832025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 6 reason = Timeout
31842025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
31852025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
31862025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 6 reason = Timeout
31872025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 6 reason = Timeout
31882025-03-21T02:06:56.917ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
31892025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
31902025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31912025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 6 reason = Timeout
31922025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 6 reason = Timeout
31932025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31942025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 6 reason = Timeout
31952025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31962025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
31972025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 7 reason = Timeout
31982025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 6 reason = Timeout
31992025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
32002025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
32012025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 6 reason = Timeout
32022025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
32032025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 6 reason = Timeout
32042025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
32052025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 6 reason = Timeout
32062025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
32072025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 6 reason = Timeout
32082025-03-21T02:06:56.917ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
32092025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
32102025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32112025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32122025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
32132025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32142025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32152025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
32162025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
32172025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
32182025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
32192025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
32202025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
32212025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
32222025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32232025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
32242025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
32252025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32262025-03-21T02:06:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
32272025-03-21T02:06:56.920ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32282025-03-21T02:06:56.921ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32292025-03-21T02:06:56.922ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
32302025-03-21T02:06:56.922ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32312025-03-21T02:06:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32322025-03-21T02:06:56.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32332025-03-21T02:06:56.922ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.17062ms iteration = 6 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
32342025-03-21T02:06:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
32352025-03-21T02:06:56.924ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32362025-03-21T02:06:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32372025-03-21T02:06:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32382025-03-21T02:06:56.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32392025-03-21T02:06:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32402025-03-21T02:06:56.928ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1125, tv_nsec: 553130153 } })
32412025-03-21T02:06:56.928ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
32422025-03-21T02:06:56.930ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32432025-03-21T02:06:56.930ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1125, tv_nsec: 555650447 } })
32442025-03-21T02:06:56.931ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
32452025-03-21T02:06:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
32462025-03-21T02:06:56.934ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
32472025-03-21T02:06:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
32482025-03-21T02:06:56.937ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32492025-03-21T02:06:56.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32502025-03-21T02:06:56.942ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
32512025-03-21T02:06:56.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
32522025-03-21T02:06:56.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32532025-03-21T02:06:56.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32542025-03-21T02:06:56.945ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32552025-03-21T02:06:56.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
32562025-03-21T02:06:56.948ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
32572025-03-21T02:06:56.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
32582025-03-21T02:06:56.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32592025-03-21T02:06:56.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32602025-03-21T02:06:56.969ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
32612025-03-21T02:06:56.969ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
32622025-03-21T02:06:56.969ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
32632025-03-21T02:06:56.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
32642025-03-21T02:06:56.973ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
32652025-03-21T02:06:56.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
32662025-03-21T02:06:56.976ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32672025-03-21T02:06:56.978ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 60.369131ms iteration = 6 status = {"errors":[],"garbage_collect_requested":[]}
32682025-03-21T02:06:56.978ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 60.513274ms iteration = 6 status = {"errors":[],"finish_invoked_ok":[]}
32692025-03-21T02:06:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
32702025-03-21T02:06:56.979ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
32712025-03-21T02:06:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
32722025-03-21T02:06:56.980ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
32732025-03-21T02:06:56.981ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 63.374653ms iteration = 6 status = {"errors":[],"requests_created_ok":[]}
32742025-03-21T02:06:56.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
32752025-03-21T02:06:56.986ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32762025-03-21T02:06:56.987ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32772025-03-21T02:06:56.988ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
32782025-03-21T02:06:56.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
32792025-03-21T02:06:56.988ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
32802025-03-21T02:06:56.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
32812025-03-21T02:06:56.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
32822025-03-21T02:06:56.989ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32832025-03-21T02:06:56.989ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
32842025-03-21T02:06:56.989ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32852025-03-21T02:06:56.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32862025-03-21T02:06:56.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
32872025-03-21T02:06:56.991ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32882025-03-21T02:06:56.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32892025-03-21T02:06:56.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32902025-03-21T02:06:56.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
32912025-03-21T02:06:56.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32922025-03-21T02:06:56.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32932025-03-21T02:06:56.994ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1125, tv_nsec: 619761183 } })
32942025-03-21T02:06:56.995ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
32952025-03-21T02:06:56.998ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
32962025-03-21T02:06:56.998ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
32972025-03-21T02:06:56.998ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "de70854d-04ad-4d4c-ae15-30dd90c8e3cf", "content-length": "26", "date": "Fri, 21 Mar 2025 02:06:56 GMT"} })
32982025-03-21T02:06:56.998ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
32992025-03-21T02:06:57.001ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
33002025-03-21T02:06:57.010ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
33012025-03-21T02:06:57.010ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
33022025-03-21T02:06:57.010ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
33032025-03-21T02:06:57.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
33042025-03-21T02:06:57.012ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 94.876461ms iteration = 16 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
33052025-03-21T02:06:57.012ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
33062025-03-21T02:06:57.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
33072025-03-21T02:06:57.015ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33082025-03-21T02:06:57.015ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
33092025-03-21T02:06:57.015ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
33102025-03-21T02:06:57.015ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
33112025-03-21T02:06:57.015ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 97.920495ms iteration = 16 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
33122025-03-21T02:06:57.015ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6118eade-4a61-46a1-84fc-b0853388c059", "content-length": "26", "date": "Fri, 21 Mar 2025 02:06:56 GMT"} })
33132025-03-21T02:06:57.015ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
33142025-03-21T02:06:57.016ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 98.869161ms iteration = 6 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
33152025-03-21T02:06:57.016ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
33162025-03-21T02:06:57.016ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
33172025-03-21T02:06:57.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
33182025-03-21T02:06:57.018ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
33192025-03-21T02:06:57.025ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 107.97851ms iteration = 6 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
33202025-03-21T02:06:57.038ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
33212025-03-21T02:06:57.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
33222025-03-21T02:06:57.039ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
33232025-03-21T02:06:57.039ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 121.740579ms iteration = 6 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
33242025-03-21T02:06:57.040ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 122.520123ms iteration = 6 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
33252025-03-21T02:06:57.056ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
33262025-03-21T02:06:57.056ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 138.770546ms iteration = 6 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
33272025-03-21T02:06:57.060ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
33282025-03-21T02:06:57.060ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
33292025-03-21T02:06:57.060ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 142.43073ms iteration = 6 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
33302025-03-21T02:06:57.063ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 145.462493ms iteration = 7 status = {"physical_disks_added":0}
33312025-03-21T02:06:57.071ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
33322025-03-21T02:06:57.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33332025-03-21T02:06:57.101ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
33342025-03-21T02:06:57.101ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
33352025-03-21T02:06:57.102ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "af035cb7-a1a9-46cb-8ce2-205f0636bc1d", "content-length": "26", "date": "Fri, 21 Mar 2025 02:06:57 GMT"} })
33362025-03-21T02:06:57.102ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
33372025-03-21T02:06:57.103ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
33382025-03-21T02:06:57.156ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
33392025-03-21T02:06:57.156ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "f584f982-36b2-4936-9def-2c9e97998551", "content-length": "1", "date": "Fri, 21 Mar 2025 02:06:56 GMT"} })
33402025-03-21T02:06:57.157ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
33412025-03-21T02:06:57.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
33422025-03-21T02:06:57.163ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 245.26891ms iteration = 6 status = {}
33432025-03-21T02:06:57.172ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
33442025-03-21T02:06:57.172ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33452025-03-21T02:06:57.202ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
33462025-03-21T02:06:57.202ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
33472025-03-21T02:06:57.204ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 286.822114ms iteration = 6 status = null
33482025-03-21T02:06:57.204ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
33492025-03-21T02:06:57.205ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33502025-03-21T02:06:57.206ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33512025-03-21T02:06:57.209ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
33522025-03-21T02:06:57.209ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
33532025-03-21T02:06:57.209ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33542025-03-21T02:06:57.214ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 296.850548ms iteration = 6 status = 0
33552025-03-21T02:06:57.266ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "25fbaa85-0649-4f2a-bd4a-bb148fb748e4", "content-length": "372", "date": "Fri, 21 Mar 2025 02:06:57 GMT"} })
33562025-03-21T02:06:57.266ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33572025-03-21T02:06:57.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33582025-03-21T02:06:57.272ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33592025-03-21T02:06:57.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33602025-03-21T02:06:57.275ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
33612025-03-21T02:06:57.275ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33622025-03-21T02:06:57.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33632025-03-21T02:06:57.306ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
33642025-03-21T02:06:57.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33652025-03-21T02:06:57.308ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33662025-03-21T02:06:57.310ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
33672025-03-21T02:06:57.310ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
33682025-03-21T02:06:57.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33692025-03-21T02:06:57.312ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33702025-03-21T02:06:57.313ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
33712025-03-21T02:06:57.313ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
33722025-03-21T02:06:57.313ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
33732025-03-21T02:06:57.313ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
33742025-03-21T02:06:57.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33752025-03-21T02:06:57.314ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "932273c0-646d-402b-8533-a5192eb57702", "date": "Fri, 21 Mar 2025 02:06:56 GMT"} })
33762025-03-21T02:06:57.314ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33772025-03-21T02:06:57.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33782025-03-21T02:06:57.318ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33792025-03-21T02:06:57.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33802025-03-21T02:06:57.323ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33812025-03-21T02:06:57.323ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33822025-03-21T02:06:57.327ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
33832025-03-21T02:06:57.327ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
33842025-03-21T02:06:57.328ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "b6f5b21d-2b5f-4fe3-8467-d24d4f74f31f", "date": "Fri, 21 Mar 2025 02:06:56 GMT"} })
33852025-03-21T02:06:57.330ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 412.610993ms iteration = 6 status = {}
33862025-03-21T02:06:57.387ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
33872025-03-21T02:06:57.387ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "22b95b90-1f3e-47bd-afce-d2401270072a", "content-length": "134", "date": "Fri, 21 Mar 2025 02:06:56 GMT"} })
33882025-03-21T02:06:57.388ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
33892025-03-21T02:06:57.388ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
33902025-03-21T02:06:57.388ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
33912025-03-21T02:06:57.388ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
33922025-03-21T02:06:57.388ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33932025-03-21T02:06:57.388ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33942025-03-21T02:06:57.388ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33952025-03-21T02:06:57.388ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
33962025-03-21T02:06:57.388ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33972025-03-21T02:06:57.388ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
33982025-03-21T02:06:57.388ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
33992025-03-21T02:06:57.388ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34002025-03-21T02:06:57.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34012025-03-21T02:06:57.392ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
34022025-03-21T02:06:57.392ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4358e09-34ec-44c8-81f3-77ab7f3ea66c", "content-length": "2", "date": "Fri, 21 Mar 2025 02:06:56 GMT"} })
34032025-03-21T02:06:57.392ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
34042025-03-21T02:06:57.392ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5c73f36-6bd6-484f-a7b6-19599286746a", "content-length": "52", "date": "Fri, 21 Mar 2025 02:06:56 GMT"} })
34052025-03-21T02:06:57.392ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
34062025-03-21T02:06:57.392ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
34072025-03-21T02:06:57.411ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
34082025-03-21T02:06:57.493ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "212874e3-f085-4436-b9d0-dd4009df8947", "date": "Fri, 21 Mar 2025 02:06:56 GMT"} })
34092025-03-21T02:06:57.493ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
34102025-03-21T02:06:57.493ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
34112025-03-21T02:06:57.493ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f881f61-445c-481d-a41b-9ebdff85d1de", "content-length": "691", "date": "Fri, 21 Mar 2025 02:06:56 GMT"} })
34122025-03-21T02:06:57.493ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
34132025-03-21T02:06:57.493ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
34142025-03-21T02:06:57.493ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
34152025-03-21T02:06:57.493ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3e83f2e-0a27-490c-a30c-e9a9ce663ade", "content-length": "682", "date": "Fri, 21 Mar 2025 02:06:57 GMT"} })
34162025-03-21T02:06:57.493ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34172025-03-21T02:06:57.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34182025-03-21T02:06:57.494ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 576.826531ms iteration = 6 status = {}
34192025-03-21T02:06:57.516ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34202025-03-21T02:06:57.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34212025-03-21T02:06:57.532ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34222025-03-21T02:06:57.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34232025-03-21T02:06:57.535ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34242025-03-21T02:06:57.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34252025-03-21T02:06:57.543ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 625.390967ms iteration = 6 status = {}
34262025-03-21T02:06:59.717ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 311 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 8652259b-93c8-4faa-a6b4-237677fdf1a4 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
34272025-03-21T02:07:06.916ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 17 reason = Timeout
34282025-03-21T02:07:06.916ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34292025-03-21T02:07:06.916ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 17 reason = Timeout
34302025-03-21T02:07:06.916ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34312025-03-21T02:07:06.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34322025-03-21T02:07:06.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34332025-03-21T02:07:06.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34342025-03-21T02:07:06.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34352025-03-21T02:07:06.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34362025-03-21T02:07:06.920ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.575713ms iteration = 17 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
34372025-03-21T02:07:06.921ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
34382025-03-21T02:07:06.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34392025-03-21T02:07:06.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34402025-03-21T02:07:06.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34412025-03-21T02:07:06.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34422025-03-21T02:07:06.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34432025-03-21T02:07:06.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
34442025-03-21T02:07:06.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34452025-03-21T02:07:06.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34462025-03-21T02:07:06.941ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
34472025-03-21T02:07:06.941ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.70697ms iteration = 17 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
34482025-03-21T02:07:09.717ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 269 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 4a1dc7df-3cf6-4f58-9831-03cbfa03af73 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
34492025-03-21T02:07:16.917ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 18 reason = Timeout
34502025-03-21T02:07:16.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34512025-03-21T02:07:16.917ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 18 reason = Timeout
34522025-03-21T02:07:16.917ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34532025-03-21T02:07:16.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34542025-03-21T02:07:16.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34552025-03-21T02:07:16.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34562025-03-21T02:07:16.923ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
34572025-03-21T02:07:16.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34582025-03-21T02:07:16.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34592025-03-21T02:07:16.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34602025-03-21T02:07:16.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34612025-03-21T02:07:16.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34622025-03-21T02:07:16.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
34632025-03-21T02:07:16.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34642025-03-21T02:07:16.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34652025-03-21T02:07:16.937ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
34662025-03-21T02:07:16.937ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.826875ms iteration = 18 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
34672025-03-21T02:07:16.948ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34682025-03-21T02:07:16.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34692025-03-21T02:07:16.977ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.773071ms iteration = 18 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
34702025-03-21T02:07:19.718ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 269 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 4900de9f-2b1a-4aaf-980b-72f9f7d24bb0 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
34712025-03-21T02:07:24.708ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:36229
34722025-03-21T02:07:24.708ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = 41905425-60cf-4e86-ad22-c750cedacae8 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
34732025-03-21T02:07:24.709ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = 41905425-60cf-4e86-ad22-c750cedacae8 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
34742025-03-21T02:07:24.712ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3280 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = 41905425-60cf-4e86-ad22-c750cedacae8 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
34752025-03-21T02:07:24.712ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = d8d99c93-54d3-4b14-91bf-fc432c3ef9a5 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
34762025-03-21T02:07:24.712ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = d8d99c93-54d3-4b14-91bf-fc432c3ef9a5 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
34772025-03-21T02:07:24.715ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3203 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = d8d99c93-54d3-4b14-91bf-fc432c3ef9a5 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
34782025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 19 reason = Timeout
34792025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34802025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 7 reason = Timeout
34812025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 7 reason = Timeout
34822025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
34832025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
34842025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 3 reason = Timeout
34852025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 4 reason = Timeout
34862025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
34872025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 7 reason = Timeout
34882025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
34892025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
34902025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
34912025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34922025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 7 reason = Timeout
34932025-03-21T02:07:26.917ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
34942025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34952025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 4 reason = Timeout
34962025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 7 reason = Timeout
34972025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34982025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 4 reason = Timeout
34992025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 7 reason = Timeout
35002025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
35012025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 7 reason = Timeout
35022025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
35032025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 4 reason = Timeout
35042025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
35052025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 8 reason = Timeout
35062025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
35072025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 4 reason = Timeout
35082025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 7 reason = Timeout
35092025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35102025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 6 reason = Timeout
35112025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
35122025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 7 reason = Timeout
35132025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 7 reason = Timeout
35142025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35152025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
35162025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 4 reason = Timeout
35172025-03-21T02:07:26.917ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 01:57:26.917907663 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
35182025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 7 reason = Timeout
35192025-03-21T02:07:26.917ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
35202025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 7 reason = Timeout
35212025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
35222025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
35232025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 4 reason = Timeout
35242025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
35252025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 4 reason = Timeout
35262025-03-21T02:07:26.918ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
35272025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
35282025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 4 reason = Timeout
35292025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
35302025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35312025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 4 reason = Timeout
35322025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
35332025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 7 reason = Timeout
35342025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
35352025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 7 reason = Timeout
35362025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 6 reason = Timeout
35372025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 5 reason = Timeout
35382025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 7 reason = Timeout
35392025-03-21T02:07:26.918ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
35402025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.04\u{b5}s iteration = 5 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
35412025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 19 reason = Timeout
35422025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
35432025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 7 reason = Timeout
35442025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
35452025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35462025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
35472025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
35482025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
35492025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35502025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35512025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35522025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35532025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35542025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
35552025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
35562025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
35572025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
35582025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35592025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
35602025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35612025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
35622025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
35632025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35642025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
35652025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
35662025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
35672025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
35682025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
35692025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
35702025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35712025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
35722025-03-21T02:07:26.918ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1155, tv_nsec: 543191952 } })
35732025-03-21T02:07:26.918ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1155, tv_nsec: 543194572 } })
35742025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35752025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35762025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
35772025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
35782025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35792025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
35802025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35812025-03-21T02:07:26.918ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
35822025-03-21T02:07:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35832025-03-21T02:07:26.919ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
35842025-03-21T02:07:26.919ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.411055ms iteration = 4 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
35852025-03-21T02:07:26.919ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
35862025-03-21T02:07:26.919ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.554727ms iteration = 4 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
35872025-03-21T02:07:26.921ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.925218ms iteration = 7 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
35882025-03-21T02:07:26.922ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1155, tv_nsec: 546646542 } })
35892025-03-21T02:07:26.922ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
35902025-03-21T02:07:26.922ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35912025-03-21T02:07:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35922025-03-21T02:07:26.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
35932025-03-21T02:07:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35942025-03-21T02:07:26.923ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35952025-03-21T02:07:26.924ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35962025-03-21T02:07:26.925ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
35972025-03-21T02:07:26.925ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35982025-03-21T02:07:26.925ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 7.4803ms iteration = 7 status = {"errors":[],"garbage_collect_requested":[]}
35992025-03-21T02:07:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36002025-03-21T02:07:26.926ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
36012025-03-21T02:07:26.926ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36022025-03-21T02:07:26.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36032025-03-21T02:07:26.927ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1155, tv_nsec: 551575428 } })
36042025-03-21T02:07:26.927ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
36052025-03-21T02:07:26.927ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36062025-03-21T02:07:26.929ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
36072025-03-21T02:07:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
36082025-03-21T02:07:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
36092025-03-21T02:07:26.930ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
36102025-03-21T02:07:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
36112025-03-21T02:07:26.932ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
36122025-03-21T02:07:26.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
36132025-03-21T02:07:26.933ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36142025-03-21T02:07:26.935ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36152025-03-21T02:07:26.936ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
36162025-03-21T02:07:26.936ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
36172025-03-21T02:07:26.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
36182025-03-21T02:07:26.936ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36192025-03-21T02:07:26.938ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
36202025-03-21T02:07:26.938ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36212025-03-21T02:07:26.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36222025-03-21T02:07:26.938ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 20.718781ms iteration = 4 status = {"errors":[],"found_port_ok":[]}
36232025-03-21T02:07:26.938ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
36242025-03-21T02:07:26.938ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36252025-03-21T02:07:26.938ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
36262025-03-21T02:07:26.938ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 01:57:26.917907663 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
36272025-03-21T02:07:26.938ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 21.018116ms iteration = 4 status = {"expiration":"2025-03-21T01:57:26.917907663Z","pruned":[]}
36282025-03-21T02:07:26.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
36292025-03-21T02:07:26.941ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
36302025-03-21T02:07:26.941ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
36312025-03-21T02:07:26.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
36322025-03-21T02:07:26.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36332025-03-21T02:07:26.942ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36342025-03-21T02:07:26.945ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
36352025-03-21T02:07:26.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36362025-03-21T02:07:26.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36372025-03-21T02:07:26.945ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
36382025-03-21T02:07:26.945ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36392025-03-21T02:07:26.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36402025-03-21T02:07:26.947ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36412025-03-21T02:07:26.948ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36422025-03-21T02:07:26.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
36432025-03-21T02:07:26.950ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36442025-03-21T02:07:26.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36452025-03-21T02:07:26.950ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36462025-03-21T02:07:26.952ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36472025-03-21T02:07:26.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
36482025-03-21T02:07:26.952ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36492025-03-21T02:07:26.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
36502025-03-21T02:07:26.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36512025-03-21T02:07:26.953ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36522025-03-21T02:07:26.955ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
36532025-03-21T02:07:26.955ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36542025-03-21T02:07:26.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
36552025-03-21T02:07:26.955ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36562025-03-21T02:07:26.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36572025-03-21T02:07:26.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36582025-03-21T02:07:26.958ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 40.479325ms iteration = 7 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
36592025-03-21T02:07:26.958ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36602025-03-21T02:07:26.959ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
36612025-03-21T02:07:26.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
36622025-03-21T02:07:26.960ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
36632025-03-21T02:07:26.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
36642025-03-21T02:07:26.961ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36652025-03-21T02:07:26.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36662025-03-21T02:07:26.963ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36672025-03-21T02:07:26.963ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
36682025-03-21T02:07:26.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36692025-03-21T02:07:26.965ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36702025-03-21T02:07:26.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
36712025-03-21T02:07:26.968ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
36722025-03-21T02:07:26.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
36732025-03-21T02:07:26.973ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
36742025-03-21T02:07:26.973ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 55.291543ms iteration = 7 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
36752025-03-21T02:07:26.984ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36762025-03-21T02:07:26.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
36772025-03-21T02:07:26.987ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36782025-03-21T02:07:26.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36792025-03-21T02:07:26.988ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
36802025-03-21T02:07:26.988ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
36812025-03-21T02:07:26.988ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
36822025-03-21T02:07:26.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
36832025-03-21T02:07:26.989ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 71.394572ms iteration = 7 status = {"errors":[],"requests_created_ok":[]}
36842025-03-21T02:07:26.991ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
36852025-03-21T02:07:26.991ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
36862025-03-21T02:07:26.991ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 73.781915ms iteration = 4 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
36872025-03-21T02:07:26.991ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 73.749144ms iteration = 7 status = {"errors":[],"finish_invoked_ok":[]}
36882025-03-21T02:07:26.992ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 74.963085ms iteration = 19 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
36892025-03-21T02:07:26.993ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
36902025-03-21T02:07:26.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
36912025-03-21T02:07:26.994ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36922025-03-21T02:07:26.995ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
36932025-03-21T02:07:26.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
36942025-03-21T02:07:26.996ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
36952025-03-21T02:07:26.996ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
36962025-03-21T02:07:26.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
36972025-03-21T02:07:27.005ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
36982025-03-21T02:07:27.006ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
36992025-03-21T02:07:27.006ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
37002025-03-21T02:07:27.006ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
37012025-03-21T02:07:27.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
37022025-03-21T02:07:27.009ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
37032025-03-21T02:07:27.009ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
37042025-03-21T02:07:27.009ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
37052025-03-21T02:07:27.009ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37062025-03-21T02:07:27.009ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
37072025-03-21T02:07:27.009ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
37082025-03-21T02:07:27.009ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 91.413691ms iteration = 4 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
37092025-03-21T02:07:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37102025-03-21T02:07:27.024ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
37112025-03-21T02:07:27.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
37122025-03-21T02:07:27.053ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
37132025-03-21T02:07:27.059ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
37142025-03-21T02:07:27.059ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
37152025-03-21T02:07:27.059ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3e5f6c9-2b00-4185-b69c-8bd274039240", "content-length": "26", "date": "Fri, 21 Mar 2025 02:07:27 GMT"} })
37162025-03-21T02:07:27.059ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
37172025-03-21T02:07:27.061ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
37182025-03-21T02:07:27.063ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
37192025-03-21T02:07:27.063ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
37202025-03-21T02:07:27.063ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a838b6e-69c3-4823-8e78-326b55da2a98", "content-length": "26", "date": "Fri, 21 Mar 2025 02:07:27 GMT"} })
37212025-03-21T02:07:27.063ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
37222025-03-21T02:07:27.064ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37232025-03-21T02:07:27.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37242025-03-21T02:07:27.065ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
37252025-03-21T02:07:27.066ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1155, tv_nsec: 691379242 } })
37262025-03-21T02:07:27.067ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
37272025-03-21T02:07:27.134ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
37282025-03-21T02:07:27.225ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
37292025-03-21T02:07:27.226ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "cef20e9f-ad01-4239-839e-eabd477bc4ad", "content-length": "1", "date": "Fri, 21 Mar 2025 02:07:26 GMT"} })
37302025-03-21T02:07:27.227ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
37312025-03-21T02:07:27.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
37322025-03-21T02:07:27.232ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
37332025-03-21T02:07:27.232ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
37342025-03-21T02:07:27.232ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
37352025-03-21T02:07:27.232ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1874349-2ec0-41fb-80ac-9bb9043159e0", "content-length": "26", "date": "Fri, 21 Mar 2025 02:07:27 GMT"} })
37362025-03-21T02:07:27.232ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
37372025-03-21T02:07:27.232ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "ab58a4d8-47b2-44bf-b676-6711acc0ec9d", "date": "Fri, 21 Mar 2025 02:07:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37382025-03-21T02:07:27.232ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37392025-03-21T02:07:27.236ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
37402025-03-21T02:07:27.237ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "f28bc502-3bf2-4fc6-a36e-d43d6da3d89a", "date": "Fri, 21 Mar 2025 02:07:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37412025-03-21T02:07:27.237ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37422025-03-21T02:07:27.239ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37432025-03-21T02:07:27.239ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 321.09214ms iteration = 6 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
37442025-03-21T02:07:27.244ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 327.137945ms iteration = 7 status = {}
37452025-03-21T02:07:27.246ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
37462025-03-21T02:07:27.246ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
37472025-03-21T02:07:27.247ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "f83f01d1-5658-4d0c-9285-0fde4d0fbee1", "content-length": "372", "date": "Fri, 21 Mar 2025 02:07:27 GMT"} })
37482025-03-21T02:07:27.247ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37492025-03-21T02:07:27.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37502025-03-21T02:07:27.248ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37512025-03-21T02:07:27.248ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37522025-03-21T02:07:27.250ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 332.666021ms iteration = 3 status = {"nsuccess":5}
37532025-03-21T02:07:27.250ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
37542025-03-21T02:07:27.250ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
37552025-03-21T02:07:27.250ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
37562025-03-21T02:07:27.251ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
37572025-03-21T02:07:27.251ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
37582025-03-21T02:07:27.251ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37592025-03-21T02:07:27.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37602025-03-21T02:07:27.252ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
37612025-03-21T02:07:27.253ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37622025-03-21T02:07:27.253ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37632025-03-21T02:07:27.253ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
37642025-03-21T02:07:27.253ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
37652025-03-21T02:07:27.254ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 336.969665ms iteration = 7 status = null
37662025-03-21T02:07:27.256ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
37672025-03-21T02:07:27.256ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
37682025-03-21T02:07:27.256ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
37692025-03-21T02:07:27.256ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37702025-03-21T02:07:27.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
37712025-03-21T02:07:27.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37722025-03-21T02:07:27.259ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37732025-03-21T02:07:27.263ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
37742025-03-21T02:07:27.263ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
37752025-03-21T02:07:27.264ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
37762025-03-21T02:07:27.267ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
37772025-03-21T02:07:27.267ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 348.996945ms iteration = 19 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
37782025-03-21T02:07:27.292ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
37792025-03-21T02:07:27.295ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
37802025-03-21T02:07:27.295ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
37812025-03-21T02:07:27.295ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
37822025-03-21T02:07:27.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
37832025-03-21T02:07:27.301ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
37842025-03-21T02:07:27.301ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
37852025-03-21T02:07:27.301ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
37862025-03-21T02:07:27.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
37872025-03-21T02:07:27.302ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
37882025-03-21T02:07:27.302ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 384.423362ms iteration = 7 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
37892025-03-21T02:07:27.303ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37902025-03-21T02:07:27.303ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 386.02818ms iteration = 7 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
37912025-03-21T02:07:27.307ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
37922025-03-21T02:07:27.307ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
37932025-03-21T02:07:27.307ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37942025-03-21T02:07:27.307ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
37952025-03-21T02:07:27.307ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
37962025-03-21T02:07:27.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37972025-03-21T02:07:27.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
37982025-03-21T02:07:27.309ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
37992025-03-21T02:07:27.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
38002025-03-21T02:07:27.309ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38012025-03-21T02:07:27.310ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
38022025-03-21T02:07:27.310ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
38032025-03-21T02:07:27.310ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-21 02:04:31.662541 UTC dns_group = external generation = 2
38042025-03-21T02:07:27.310ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 392.801947ms iteration = 4 status = {"generation":2}
38052025-03-21T02:07:27.310ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38062025-03-21T02:07:27.312ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 394.772942ms iteration = 7 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
38072025-03-21T02:07:27.312ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
38082025-03-21T02:07:27.312ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
38092025-03-21T02:07:27.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
38102025-03-21T02:07:27.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
38112025-03-21T02:07:27.313ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
38122025-03-21T02:07:27.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
38132025-03-21T02:07:27.316ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
38142025-03-21T02:07:27.316ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
38152025-03-21T02:07:27.316ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-21 02:04:30.886360 UTC dns_group = internal generation = 1
38162025-03-21T02:07:27.316ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 398.524597ms iteration = 4 status = {"generation":1}
38172025-03-21T02:07:27.320ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
38182025-03-21T02:07:27.320ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
38192025-03-21T02:07:27.320ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "0992e01e-064e-48e4-8f0b-5f3d7c8e9e64", "date": "Fri, 21 Mar 2025 02:07:27 GMT"} })
38202025-03-21T02:07:27.320ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38212025-03-21T02:07:27.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38222025-03-21T02:07:27.323ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38232025-03-21T02:07:27.323ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38242025-03-21T02:07:27.326ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38252025-03-21T02:07:27.326ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38262025-03-21T02:07:27.329ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
38272025-03-21T02:07:27.329ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
38282025-03-21T02:07:27.333ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "d1c65324-e3c7-4a1b-a54f-288dae0c5566", "date": "Fri, 21 Mar 2025 02:07:27 GMT"} })
38292025-03-21T02:07:27.334ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 416.942338ms iteration = 7 status = {}
38302025-03-21T02:07:27.337ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38312025-03-21T02:07:27.337ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
38322025-03-21T02:07:27.337ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 419.492332ms iteration = 7 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
38332025-03-21T02:07:27.344ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38342025-03-21T02:07:27.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38352025-03-21T02:07:27.349ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) }
38362025-03-21T02:07:27.349ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 432.03142ms iteration = 4 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
38372025-03-21T02:07:27.368ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
38382025-03-21T02:07:27.368ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "a7f7c300-4d0e-44d5-af25-a5ccc5486e07", "date": "Fri, 21 Mar 2025 02:07:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
38392025-03-21T02:07:27.368ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
38402025-03-21T02:07:27.370ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "6f2b2814-6f2e-4781-8f9f-f85d88b5913d", "date": "Fri, 21 Mar 2025 02:07:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
38412025-03-21T02:07:27.370ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
38422025-03-21T02:07:27.370ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 452.873423ms iteration = 8 status = {"physical_disks_added":0}
38432025-03-21T02:07:27.371ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
38442025-03-21T02:07:27.371ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 453.24764ms iteration = 4 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
38452025-03-21T02:07:27.372ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "51e58680-429d-42cb-8bb5-8a9c6eb147bc", "date": "Fri, 21 Mar 2025 02:07:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
38462025-03-21T02:07:27.372ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
38472025-03-21T02:07:27.374ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
38482025-03-21T02:07:27.374ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 456.415815ms iteration = 6 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
38492025-03-21T02:07:27.377ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38502025-03-21T02:07:27.378ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
38512025-03-21T02:07:27.378ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38522025-03-21T02:07:27.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38532025-03-21T02:07:27.380ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38542025-03-21T02:07:27.382ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
38552025-03-21T02:07:27.382ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38562025-03-21T02:07:27.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38572025-03-21T02:07:27.441ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38582025-03-21T02:07:27.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38592025-03-21T02:07:27.443ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
38602025-03-21T02:07:27.443ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38612025-03-21T02:07:27.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38622025-03-21T02:07:27.458ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
38632025-03-21T02:07:27.458ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "937fab95-e4dc-42fe-8072-a5268fb69642", "content-length": "691", "date": "Fri, 21 Mar 2025 02:07:27 GMT"} })
38642025-03-21T02:07:27.458ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
38652025-03-21T02:07:27.458ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
38662025-03-21T02:07:27.458ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
38672025-03-21T02:07:27.459ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 542.220339ms iteration = 7 status = {}
38682025-03-21T02:07:27.558ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
38692025-03-21T02:07:27.559ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c7c9881-9399-4026-aaf0-502407988293", "content-length": "134", "date": "Fri, 21 Mar 2025 02:07:27 GMT"} })
38702025-03-21T02:07:27.559ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
38712025-03-21T02:07:27.559ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
38722025-03-21T02:07:27.559ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
38732025-03-21T02:07:27.559ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
38742025-03-21T02:07:27.559ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
38752025-03-21T02:07:27.559ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
38762025-03-21T02:07:27.559ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
38772025-03-21T02:07:27.559ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
38782025-03-21T02:07:27.559ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
38792025-03-21T02:07:27.559ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
38802025-03-21T02:07:27.559ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
38812025-03-21T02:07:27.559ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38822025-03-21T02:07:27.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38832025-03-21T02:07:27.562ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
38842025-03-21T02:07:27.562ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc18f852-d78f-4ebd-aa37-0f06d7ce331a", "content-length": "2", "date": "Fri, 21 Mar 2025 02:07:27 GMT"} })
38852025-03-21T02:07:27.562ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
38862025-03-21T02:07:27.562ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "338e6f93-9dad-435a-bea3-4eb6799c2d35", "content-length": "52", "date": "Fri, 21 Mar 2025 02:07:27 GMT"} })
38872025-03-21T02:07:27.562ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
38882025-03-21T02:07:27.562ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
38892025-03-21T02:07:27.574ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38902025-03-21T02:07:27.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38912025-03-21T02:07:27.576ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38922025-03-21T02:07:27.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38932025-03-21T02:07:27.577ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38942025-03-21T02:07:27.580ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
38952025-03-21T02:07:27.580ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38962025-03-21T02:07:27.580ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38972025-03-21T02:07:27.581ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 663.747404ms iteration = 7 status = 0
38982025-03-21T02:07:27.657ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "22e1b8f2-c0d6-4977-bdf8-dd784ea31e64", "date": "Fri, 21 Mar 2025 02:07:27 GMT"} })
38992025-03-21T02:07:27.657ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
39002025-03-21T02:07:27.657ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
39012025-03-21T02:07:27.657ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "590b31b7-fae7-43ec-a6bb-2556ef23eab0", "content-length": "682", "date": "Fri, 21 Mar 2025 02:07:27 GMT"} })
39022025-03-21T02:07:27.657ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39032025-03-21T02:07:27.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39042025-03-21T02:07:27.661ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39052025-03-21T02:07:27.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39062025-03-21T02:07:27.665ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39072025-03-21T02:07:27.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39082025-03-21T02:07:27.666ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39092025-03-21T02:07:27.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39102025-03-21T02:07:27.674ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 756.371227ms iteration = 7 status = {}
39112025-03-21T02:07:29.718ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 310 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 8cbf9ff9-ee29-475e-bdc6-45f49a0d7ac8 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
39122025-03-21T02:07:36.917ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 20 reason = Timeout
39132025-03-21T02:07:36.917ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 20 reason = Timeout
39142025-03-21T02:07:36.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39152025-03-21T02:07:36.917ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39162025-03-21T02:07:36.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39172025-03-21T02:07:36.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39182025-03-21T02:07:36.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39192025-03-21T02:07:36.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39202025-03-21T02:07:36.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39212025-03-21T02:07:36.922ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
39222025-03-21T02:07:36.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39232025-03-21T02:07:36.922ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.491169ms iteration = 20 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
39242025-03-21T02:07:36.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39252025-03-21T02:07:36.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39262025-03-21T02:07:36.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39272025-03-21T02:07:36.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39282025-03-21T02:07:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
39292025-03-21T02:07:36.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39302025-03-21T02:07:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39312025-03-21T02:07:37.049ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
39322025-03-21T02:07:37.049ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 131.222294ms iteration = 20 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
39332025-03-21T02:07:39.718ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 302 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 18f185cf-f2cc-4e97-b192-df6bc40804c6 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
39342025-03-21T02:07:46.917ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 21 reason = Timeout
39352025-03-21T02:07:46.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39362025-03-21T02:07:46.917ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 21 reason = Timeout
39372025-03-21T02:07:46.917ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39382025-03-21T02:07:46.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39392025-03-21T02:07:46.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39402025-03-21T02:07:46.919ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39412025-03-21T02:07:46.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39422025-03-21T02:07:46.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39432025-03-21T02:07:46.920ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.434719ms iteration = 21 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
39442025-03-21T02:07:46.921ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
39452025-03-21T02:07:46.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39462025-03-21T02:07:46.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39472025-03-21T02:07:46.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39482025-03-21T02:07:46.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39492025-03-21T02:07:46.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39502025-03-21T02:07:46.926ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
39512025-03-21T02:07:46.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39522025-03-21T02:07:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39532025-03-21T02:07:46.936ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
39542025-03-21T02:07:46.936ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.736736ms iteration = 21 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
39552025-03-21T02:07:49.718ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 320 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 982bcb3c-fb70-42dc-a98f-0d8b84954edf response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
39562025-03-21T02:07:54.709ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = d9b5ca99-061d-4799-aacb-612e49d263ba roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
39572025-03-21T02:07:54.709ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = d9b5ca99-061d-4799-aacb-612e49d263ba resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
39582025-03-21T02:07:54.712ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3114 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = d9b5ca99-061d-4799-aacb-612e49d263ba response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
39592025-03-21T02:07:54.712ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = aee1bb96-880e-467c-9ec3-2eda1f5fb3d3 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
39602025-03-21T02:07:54.712ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = aee1bb96-880e-467c-9ec3-2eda1f5fb3d3 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
39612025-03-21T02:07:54.714ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2399 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = aee1bb96-880e-467c-9ec3-2eda1f5fb3d3 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
39622025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 22 reason = Timeout
39632025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39642025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 8 reason = Timeout
39652025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 8 reason = Timeout
39662025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 8 reason = Timeout
39672025-03-21T02:07:56.918ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
39682025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 22 reason = Timeout
39692025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39702025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 8 reason = Timeout
39712025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39722025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
39732025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39742025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 8 reason = Timeout
39752025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 8 reason = Timeout
39762025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
39772025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
39782025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 8 reason = Timeout
39792025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
39802025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 8 reason = Timeout
39812025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
39822025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 8 reason = Timeout
39832025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
39842025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 8 reason = Timeout
39852025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
39862025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 9 reason = Timeout
39872025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 8 reason = Timeout
39882025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
39892025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
39902025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 8 reason = Timeout
39912025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
39922025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 8 reason = Timeout
39932025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
39942025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
39952025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 8 reason = Timeout
39962025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39972025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 8 reason = Timeout
39982025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39992025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 8 reason = Timeout
40002025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
40012025-03-21T02:07:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40022025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40032025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40042025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
40052025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
40062025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
40072025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
40082025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
40092025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
40102025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
40112025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
40122025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
40132025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
40142025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40152025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
40162025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40172025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
40182025-03-21T02:07:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
40192025-03-21T02:07:56.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40202025-03-21T02:07:56.921ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.534885ms iteration = 8 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
40212025-03-21T02:07:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40222025-03-21T02:07:56.921ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40232025-03-21T02:07:56.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40242025-03-21T02:07:56.922ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40252025-03-21T02:07:56.922ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40262025-03-21T02:07:56.923ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
40272025-03-21T02:07:56.923ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40282025-03-21T02:07:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40292025-03-21T02:07:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
40302025-03-21T02:07:56.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40312025-03-21T02:07:56.924ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 5.962414ms iteration = 8 status = {"errors":[],"finish_invoked_ok":[]}
40322025-03-21T02:07:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40332025-03-21T02:07:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
40342025-03-21T02:07:56.925ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
40352025-03-21T02:07:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
40362025-03-21T02:07:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
40372025-03-21T02:07:56.925ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
40382025-03-21T02:07:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
40392025-03-21T02:07:56.927ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1185, tv_nsec: 551413467 } })
40402025-03-21T02:07:56.927ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1185, tv_nsec: 551549730 } })
40412025-03-21T02:07:56.927ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
40422025-03-21T02:07:56.927ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40432025-03-21T02:07:56.927ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
40442025-03-21T02:07:56.927ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
40452025-03-21T02:07:56.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
40462025-03-21T02:07:56.928ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40472025-03-21T02:07:56.928ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40482025-03-21T02:07:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
40492025-03-21T02:07:56.930ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
40502025-03-21T02:07:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
40512025-03-21T02:07:56.930ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
40522025-03-21T02:07:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
40532025-03-21T02:07:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
40542025-03-21T02:07:56.932ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
40552025-03-21T02:07:56.932ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40562025-03-21T02:07:56.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40572025-03-21T02:07:56.934ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 16.191402ms iteration = 22 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
40582025-03-21T02:07:56.935ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40592025-03-21T02:07:56.937ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40602025-03-21T02:07:56.937ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
40612025-03-21T02:07:56.937ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40622025-03-21T02:07:56.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40632025-03-21T02:07:56.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40642025-03-21T02:07:56.937ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
40652025-03-21T02:07:56.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
40662025-03-21T02:07:56.942ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
40672025-03-21T02:07:56.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
40682025-03-21T02:07:56.943ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1185, tv_nsec: 567537748 } })
40692025-03-21T02:07:56.943ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
40702025-03-21T02:07:56.947ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40712025-03-21T02:07:56.949ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
40722025-03-21T02:07:56.949ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
40732025-03-21T02:07:56.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
40742025-03-21T02:07:56.963ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 45.318669ms iteration = 8 status = {"errors":[],"garbage_collect_requested":[]}
40752025-03-21T02:07:56.971ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
40762025-03-21T02:07:56.972ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
40772025-03-21T02:07:56.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
40782025-03-21T02:07:56.972ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
40792025-03-21T02:07:56.972ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
40802025-03-21T02:07:56.972ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
40812025-03-21T02:07:56.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
40822025-03-21T02:07:56.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
40832025-03-21T02:07:56.974ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40842025-03-21T02:07:56.974ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
40852025-03-21T02:07:56.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
40862025-03-21T02:07:56.974ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
40872025-03-21T02:07:56.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
40882025-03-21T02:07:56.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
40892025-03-21T02:07:56.975ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
40902025-03-21T02:07:56.976ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
40912025-03-21T02:07:56.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
40922025-03-21T02:07:56.976ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
40932025-03-21T02:07:56.976ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
40942025-03-21T02:07:56.976ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 57.401339ms iteration = 8 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
40952025-03-21T02:07:56.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
40962025-03-21T02:07:56.977ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
40972025-03-21T02:07:56.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
40982025-03-21T02:07:56.979ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40992025-03-21T02:07:56.980ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
41002025-03-21T02:07:56.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
41012025-03-21T02:07:56.982ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 63.853851ms iteration = 8 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
41022025-03-21T02:07:56.983ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
41032025-03-21T02:07:56.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41042025-03-21T02:07:56.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41052025-03-21T02:07:56.985ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 66.908734ms iteration = 8 status = 0
41062025-03-21T02:07:56.985ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
41072025-03-21T02:07:56.985ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 66.983686ms iteration = 8 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
41082025-03-21T02:07:56.988ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 70.212962ms iteration = 8 status = {"errors":[],"requests_created_ok":[]}
41092025-03-21T02:07:56.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41102025-03-21T02:07:56.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
41112025-03-21T02:07:56.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41122025-03-21T02:07:56.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41132025-03-21T02:07:56.993ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
41142025-03-21T02:07:56.993ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
41152025-03-21T02:07:56.994ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "34dcdcee-840b-4ec1-adeb-47ebc65e2605", "content-length": "26", "date": "Fri, 21 Mar 2025 02:07:56 GMT"} })
41162025-03-21T02:07:56.994ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
41172025-03-21T02:07:56.996ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
41182025-03-21T02:07:57.001ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
41192025-03-21T02:07:57.001ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 83.25631ms iteration = 22 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
41202025-03-21T02:07:57.004ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 85.927106ms iteration = 8 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
41212025-03-21T02:07:57.009ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
41222025-03-21T02:07:57.009ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
41232025-03-21T02:07:57.009ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d8e13fd-26c2-49ac-a52b-f91c38a8336f", "content-length": "26", "date": "Fri, 21 Mar 2025 02:07:56 GMT"} })
41242025-03-21T02:07:57.009ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
41252025-03-21T02:07:57.011ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
41262025-03-21T02:07:57.012ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
41272025-03-21T02:07:57.012ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 94.261211ms iteration = 8 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
41282025-03-21T02:07:57.020ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
41292025-03-21T02:07:57.020ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
41302025-03-21T02:07:57.020ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 101.996356ms iteration = 8 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
41312025-03-21T02:07:57.043ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
41322025-03-21T02:07:57.043ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
41332025-03-21T02:07:57.043ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0a87dac-0ff9-46d1-b2c5-7719a02ff495", "content-length": "26", "date": "Fri, 21 Mar 2025 02:07:56 GMT"} })
41342025-03-21T02:07:57.043ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
41352025-03-21T02:07:57.045ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
41362025-03-21T02:07:57.047ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
41372025-03-21T02:07:57.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
41382025-03-21T02:07:57.050ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 131.47926ms iteration = 9 status = {"physical_disks_added":0}
41392025-03-21T02:07:57.082ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
41402025-03-21T02:07:57.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
41412025-03-21T02:07:57.084ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41422025-03-21T02:07:57.086ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
41432025-03-21T02:07:57.086ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
41442025-03-21T02:07:57.086ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
41452025-03-21T02:07:57.087ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41462025-03-21T02:07:57.088ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 170.233934ms iteration = 8 status = {}
41472025-03-21T02:07:57.089ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
41482025-03-21T02:07:57.089ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
41492025-03-21T02:07:57.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
41502025-03-21T02:07:57.093ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
41512025-03-21T02:07:57.093ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "d99b4873-bdf8-4da8-80cd-56148e4d0cdf", "content-length": "1", "date": "Fri, 21 Mar 2025 02:07:57 GMT"} })
41522025-03-21T02:07:57.095ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
41532025-03-21T02:07:57.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
41542025-03-21T02:07:57.097ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 178.701561ms iteration = 8 status = null
41552025-03-21T02:07:57.103ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
41562025-03-21T02:07:57.103ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
41572025-03-21T02:07:57.104ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "90c1dce5-dda7-41b9-a147-e3814e7a54e5", "content-length": "372", "date": "Fri, 21 Mar 2025 02:07:56 GMT"} })
41582025-03-21T02:07:57.104ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41592025-03-21T02:07:57.104ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41602025-03-21T02:07:57.118ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
41612025-03-21T02:07:57.118ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
41622025-03-21T02:07:57.118ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "af126001-0f44-40f2-94ee-9be8e43e1f9b", "date": "Fri, 21 Mar 2025 02:07:56 GMT"} })
41632025-03-21T02:07:57.118ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41642025-03-21T02:07:57.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41652025-03-21T02:07:57.120ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41662025-03-21T02:07:57.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41672025-03-21T02:07:57.123ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41682025-03-21T02:07:57.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41692025-03-21T02:07:57.160ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
41702025-03-21T02:07:57.163ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
41712025-03-21T02:07:57.163ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
41722025-03-21T02:07:57.207ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41732025-03-21T02:07:57.207ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41742025-03-21T02:07:57.209ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
41752025-03-21T02:07:57.209ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41762025-03-21T02:07:57.209ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41772025-03-21T02:07:57.218ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
41782025-03-21T02:07:57.218ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "495d7b19-018d-47a6-9437-ab81b991b061", "content-length": "134", "date": "Fri, 21 Mar 2025 02:07:57 GMT"} })
41792025-03-21T02:07:57.218ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
41802025-03-21T02:07:57.218ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
41812025-03-21T02:07:57.218ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
41822025-03-21T02:07:57.218ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
41832025-03-21T02:07:57.218ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
41842025-03-21T02:07:57.218ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
41852025-03-21T02:07:57.218ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
41862025-03-21T02:07:57.218ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
41872025-03-21T02:07:57.218ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
41882025-03-21T02:07:57.218ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
41892025-03-21T02:07:57.219ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
41902025-03-21T02:07:57.219ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41912025-03-21T02:07:57.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41922025-03-21T02:07:57.221ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
41932025-03-21T02:07:57.221ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e6ecda8-0676-45a9-aa18-69ccd536d9d4", "content-length": "2", "date": "Fri, 21 Mar 2025 02:07:57 GMT"} })
41942025-03-21T02:07:57.221ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
41952025-03-21T02:07:57.221ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "81960b83-5c7a-4d9a-b034-3fd0bd368141", "content-length": "52", "date": "Fri, 21 Mar 2025 02:07:57 GMT"} })
41962025-03-21T02:07:57.221ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
41972025-03-21T02:07:57.221ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
41982025-03-21T02:07:57.239ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "1db934ae-d2f1-4616-8e3b-03d7cbaa32fe", "date": "Fri, 21 Mar 2025 02:07:56 GMT"} })
41992025-03-21T02:07:57.240ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 322.288731ms iteration = 8 status = {}
42002025-03-21T02:07:57.240ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb72bd86-220d-4e76-9754-0142e564a707", "content-length": "691", "date": "Fri, 21 Mar 2025 02:07:57 GMT"} })
42012025-03-21T02:07:57.240ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
42022025-03-21T02:07:57.240ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
42032025-03-21T02:07:57.240ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
42042025-03-21T02:07:57.242ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 324.070531ms iteration = 8 status = {}
42052025-03-21T02:07:57.425ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "9bba9786-999c-4a84-9fca-71f9682ee1c3", "date": "Fri, 21 Mar 2025 02:07:57 GMT"} })
42062025-03-21T02:07:57.425ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
42072025-03-21T02:07:57.425ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
42082025-03-21T02:07:57.426ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "267cb584-48ee-4a99-be91-6aac82f2eb25", "content-length": "682", "date": "Fri, 21 Mar 2025 02:07:57 GMT"} })
42092025-03-21T02:07:57.426ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42102025-03-21T02:07:57.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42112025-03-21T02:07:57.428ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42122025-03-21T02:07:57.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42132025-03-21T02:07:57.431ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42142025-03-21T02:07:57.432ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42152025-03-21T02:07:57.434ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42162025-03-21T02:07:57.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42172025-03-21T02:07:57.440ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 522.132699ms iteration = 8 status = {}
42182025-03-21T02:07:57.642ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
42192025-03-21T02:07:57.642ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
42202025-03-21T02:07:57.672ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
42212025-03-21T02:07:57.672ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:38723
42222025-03-21T02:07:57.672ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:38723 req_id = dfb501ec-c456-47bd-b16a-65b724ae163d roles = RoleSet { roles: {} } uri = /metrics/producers
42232025-03-21T02:07:57.672ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:38723 req_id = dfb501ec-c456-47bd-b16a-65b724ae163d resource = Database result = Ok(()) uri = /metrics/producers
42242025-03-21T02:07:57.714ZINFOnexus: assigned collector to new producer collector_id = a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53 file = nexus/src/app/oximeter.rs:76 producer_id = e05eb799-8cf0-4702-bd3b-9994ae86118a
42252025-03-21T02:07:57.714ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 42570 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:38723 req_id = dfb501ec-c456-47bd-b16a-65b724ae163d response_code = 201 uri = /metrics/producers
42262025-03-21T02:07:57.714ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "dfb501ec-c456-47bd-b16a-65b724ae163d", "content-length": "41", "date": "Fri, 21 Mar 2025 02:07:57 GMT"} })
42272025-03-21T02:07:57.715ZDEBGnexus (producer-registration-task): registered with nexus successfully
42282025-03-21T02:07:57.715ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
42292025-03-21T02:07:59.718ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 294 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = ff90e269-9474-4385-b9bb-4b5a656afdce response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
42302025-03-21T02:08:06.917ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 23 reason = Timeout
42312025-03-21T02:08:06.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
42322025-03-21T02:08:06.917ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 23 reason = Timeout
42332025-03-21T02:08:06.917ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42342025-03-21T02:08:06.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42352025-03-21T02:08:06.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
42362025-03-21T02:08:06.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42372025-03-21T02:08:06.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
42382025-03-21T02:08:06.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
42392025-03-21T02:08:06.922ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.252094ms iteration = 23 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
42402025-03-21T02:08:06.922ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
42412025-03-21T02:08:06.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42422025-03-21T02:08:06.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42432025-03-21T02:08:06.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42442025-03-21T02:08:06.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42452025-03-21T02:08:06.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42462025-03-21T02:08:06.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
42472025-03-21T02:08:06.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42482025-03-21T02:08:06.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42492025-03-21T02:08:06.934ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
42502025-03-21T02:08:06.934ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.922505ms iteration = 23 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
42512025-03-21T02:08:09.718ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 291 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 3901cac7-9dc6-4739-809c-55774aaa6bad response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
42522025-03-21T02:08:16.917ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 24 reason = Timeout
42532025-03-21T02:08:16.917ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
42542025-03-21T02:08:16.917ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 24 reason = Timeout
42552025-03-21T02:08:16.917ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42562025-03-21T02:08:16.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42572025-03-21T02:08:16.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
42582025-03-21T02:08:16.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42592025-03-21T02:08:16.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
42602025-03-21T02:08:16.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
42612025-03-21T02:08:16.922ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
42622025-03-21T02:08:16.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42632025-03-21T02:08:16.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42642025-03-21T02:08:16.922ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.742553ms iteration = 24 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
42652025-03-21T02:08:16.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42662025-03-21T02:08:16.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42672025-03-21T02:08:16.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42682025-03-21T02:08:16.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
42692025-03-21T02:08:16.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42702025-03-21T02:08:16.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42712025-03-21T02:08:16.935ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
42722025-03-21T02:08:16.935ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.81862ms iteration = 24 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
42732025-03-21T02:08:19.718ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 309 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 93d0f0ab-0a64-4d9f-9970-0dcf63f65cea response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
42742025-03-21T02:08:24.709ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = 3e551117-895a-4b06-a410-e817819d5020 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
42752025-03-21T02:08:24.710ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = 3e551117-895a-4b06-a410-e817819d5020 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
42762025-03-21T02:08:24.713ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3402 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = 3e551117-895a-4b06-a410-e817819d5020 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
42772025-03-21T02:08:24.713ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = 3b63de8a-014d-4ef8-a9e9-7ed8f31d5477 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
42782025-03-21T02:08:24.713ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = 3b63de8a-014d-4ef8-a9e9-7ed8f31d5477 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
42792025-03-21T02:08:24.716ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2747 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = 3b63de8a-014d-4ef8-a9e9-7ed8f31d5477 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
42802025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 25 reason = Timeout
42812025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
42822025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 9 reason = Timeout
42832025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
42842025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 6 reason = Timeout
42852025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 5 reason = Timeout
42862025-03-21T02:08:26.918ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
42872025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
42882025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 20.97\u{b5}s iteration = 6 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
42892025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 9 reason = Timeout
42902025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 9 reason = Timeout
42912025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
42922025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
42932025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 9 reason = Timeout
42942025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
42952025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 5 reason = Timeout
42962025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
42972025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 7 reason = Timeout
42982025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 25 reason = Timeout
42992025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 5 reason = Timeout
43002025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43012025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
43022025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 9 reason = Timeout
43032025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
43042025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
43052025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 5 reason = Timeout
43062025-03-21T02:08:26.918ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
43072025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 5 reason = Timeout
43082025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
43092025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 9 reason = Timeout
43102025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
43112025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 5 reason = Timeout
43122025-03-21T02:08:26.918ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 01:58:26.918427342 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
43132025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
43142025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 9 reason = Timeout
43152025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 5 reason = Timeout
43162025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
43172025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 7 reason = Timeout
43182025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 5 reason = Timeout
43192025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
43202025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 9 reason = Timeout
43212025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
43222025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 9 reason = Timeout
43232025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
43242025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 9 reason = Timeout
43252025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 5 reason = Timeout
43262025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
43272025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 10 reason = Timeout
43282025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
43292025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 9 reason = Timeout
43302025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 5 reason = Timeout
43312025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
43322025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
43332025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 9 reason = Timeout
43342025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
43352025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
43362025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 9 reason = Timeout
43372025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
43382025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
43392025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 9 reason = Timeout
43402025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43412025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 9 reason = Timeout
43422025-03-21T02:08:26.918ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
43432025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43442025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 9 reason = Timeout
43452025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
43462025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
43472025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43482025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
43492025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
43502025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43512025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
43522025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
43532025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
43542025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
43552025-03-21T02:08:26.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
43562025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
43572025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
43582025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
43592025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
43602025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
43612025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
43622025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43632025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
43642025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
43652025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
43662025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
43672025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
43682025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
43692025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
43702025-03-21T02:08:26.919ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1215, tv_nsec: 543084782 } })
43712025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43722025-03-21T02:08:26.919ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1215, tv_nsec: 543104893 } })
43732025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
43742025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
43752025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.261132ms iteration = 5 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
43762025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
43772025-03-21T02:08:26.919ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.209371ms iteration = 5 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
43782025-03-21T02:08:26.921ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1215, tv_nsec: 545345492 } })
43792025-03-21T02:08:26.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43802025-03-21T02:08:26.921ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
43812025-03-21T02:08:26.922ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.315985ms iteration = 9 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
43822025-03-21T02:08:26.923ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 4.769643ms iteration = 9 status = {"errors":[],"finish_invoked_ok":[]}
43832025-03-21T02:08:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
43842025-03-21T02:08:26.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43852025-03-21T02:08:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43862025-03-21T02:08:26.926ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
43872025-03-21T02:08:26.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
43882025-03-21T02:08:26.927ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 9.18707ms iteration = 5 status = {"errors":[],"found_port_ok":[]}
43892025-03-21T02:08:26.927ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43902025-03-21T02:08:26.928ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43912025-03-21T02:08:26.928ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43922025-03-21T02:08:26.930ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1215, tv_nsec: 553951912 } })
43932025-03-21T02:08:26.930ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
43942025-03-21T02:08:26.930ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
43952025-03-21T02:08:26.930ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
43962025-03-21T02:08:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
43972025-03-21T02:08:26.931ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 01:58:26.918427342 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
43982025-03-21T02:08:26.931ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 12.597509ms iteration = 5 status = {"expiration":"2025-03-21T01:58:26.918427342Z","pruned":[]}
43992025-03-21T02:08:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
44002025-03-21T02:08:26.931ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
44012025-03-21T02:08:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
44022025-03-21T02:08:26.931ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
44032025-03-21T02:08:26.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
44042025-03-21T02:08:26.932ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
44052025-03-21T02:08:26.932ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44062025-03-21T02:08:26.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44072025-03-21T02:08:26.941ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44082025-03-21T02:08:26.942ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44092025-03-21T02:08:26.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
44102025-03-21T02:08:26.944ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44112025-03-21T02:08:26.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
44122025-03-21T02:08:26.944ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
44132025-03-21T02:08:26.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44142025-03-21T02:08:26.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
44152025-03-21T02:08:26.946ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 27.661541ms iteration = 9 status = {"errors":[],"requests_created_ok":[]}
44162025-03-21T02:08:26.947ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44172025-03-21T02:08:26.948ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
44182025-03-21T02:08:26.948ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
44192025-03-21T02:08:26.948ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 29.571275ms iteration = 5 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
44202025-03-21T02:08:26.948ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44212025-03-21T02:08:26.949ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
44222025-03-21T02:08:26.949ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
44232025-03-21T02:08:26.949ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
44242025-03-21T02:08:26.949ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
44252025-03-21T02:08:26.949ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
44262025-03-21T02:08:26.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
44272025-03-21T02:08:26.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44282025-03-21T02:08:26.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44292025-03-21T02:08:26.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
44302025-03-21T02:08:26.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
44312025-03-21T02:08:26.950ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
44322025-03-21T02:08:26.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
44332025-03-21T02:08:26.951ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
44342025-03-21T02:08:26.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
44352025-03-21T02:08:26.952ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
44362025-03-21T02:08:26.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
44372025-03-21T02:08:26.952ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44382025-03-21T02:08:26.952ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44392025-03-21T02:08:26.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44402025-03-21T02:08:26.954ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
44412025-03-21T02:08:26.954ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1215, tv_nsec: 578379637 } })
44422025-03-21T02:08:26.954ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
44432025-03-21T02:08:26.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
44442025-03-21T02:08:26.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
44452025-03-21T02:08:26.954ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
44462025-03-21T02:08:26.957ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
44472025-03-21T02:08:26.961ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 43.289954ms iteration = 9 status = {"errors":[],"garbage_collect_requested":[]}
44482025-03-21T02:08:26.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44492025-03-21T02:08:26.963ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
44502025-03-21T02:08:26.963ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 45.197667ms iteration = 9 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
44512025-03-21T02:08:26.964ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44522025-03-21T02:08:26.966ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
44532025-03-21T02:08:26.966ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
44542025-03-21T02:08:26.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
44552025-03-21T02:08:26.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
44562025-03-21T02:08:26.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44572025-03-21T02:08:26.967ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44582025-03-21T02:08:26.967ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44592025-03-21T02:08:26.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44602025-03-21T02:08:26.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
44612025-03-21T02:08:26.970ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
44622025-03-21T02:08:26.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
44632025-03-21T02:08:26.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
44642025-03-21T02:08:26.970ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
44652025-03-21T02:08:26.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
44662025-03-21T02:08:26.972ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44672025-03-21T02:08:26.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44682025-03-21T02:08:26.978ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
44692025-03-21T02:08:26.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
44702025-03-21T02:08:26.980ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.451437ms iteration = 25 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
44712025-03-21T02:08:26.990ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
44722025-03-21T02:08:26.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
44732025-03-21T02:08:26.992ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44742025-03-21T02:08:26.992ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
44752025-03-21T02:08:26.992ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44762025-03-21T02:08:26.993ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
44772025-03-21T02:08:26.993ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 74.732751ms iteration = 25 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
44782025-03-21T02:08:26.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
44792025-03-21T02:08:26.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
44802025-03-21T02:08:26.995ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
44812025-03-21T02:08:26.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
44822025-03-21T02:08:26.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
44832025-03-21T02:08:27.006ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
44842025-03-21T02:08:27.006ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
44852025-03-21T02:08:27.006ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
44862025-03-21T02:08:27.006ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
44872025-03-21T02:08:27.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
44882025-03-21T02:08:27.008ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 90.221661ms iteration = 9 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
44892025-03-21T02:08:27.010ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
44902025-03-21T02:08:27.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
44912025-03-21T02:08:27.013ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 94.80953ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
44922025-03-21T02:08:27.014ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
44932025-03-21T02:08:27.014ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
44942025-03-21T02:08:27.014ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
44952025-03-21T02:08:27.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
44962025-03-21T02:08:27.021ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
44972025-03-21T02:08:27.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
44982025-03-21T02:08:27.024ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
44992025-03-21T02:08:27.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45002025-03-21T02:08:27.025ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 107.698935ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
45012025-03-21T02:08:27.028ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
45022025-03-21T02:08:27.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
45032025-03-21T02:08:27.039ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
45042025-03-21T02:08:27.039ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
45052025-03-21T02:08:27.039ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
45062025-03-21T02:08:27.039ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
45072025-03-21T02:08:27.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
45082025-03-21T02:08:27.052ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
45092025-03-21T02:08:27.060ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
45102025-03-21T02:08:27.060ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
45112025-03-21T02:08:27.060ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 141.89985ms iteration = 5 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
45122025-03-21T02:08:27.083ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 165.318558ms iteration = 10 status = {"physical_disks_added":0}
45132025-03-21T02:08:27.084ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
45142025-03-21T02:08:27.084ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
45152025-03-21T02:08:27.084ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a8e085c-6b68-4612-ade0-aef42bfb2484", "content-length": "26", "date": "Fri, 21 Mar 2025 02:08:26 GMT"} })
45162025-03-21T02:08:27.084ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
45172025-03-21T02:08:27.087ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
45182025-03-21T02:08:27.136ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
45192025-03-21T02:08:27.136ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
45202025-03-21T02:08:27.136ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceabccec-7b92-4267-83a9-81bf3a8b6875", "content-length": "26", "date": "Fri, 21 Mar 2025 02:08:26 GMT"} })
45212025-03-21T02:08:27.136ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
45222025-03-21T02:08:27.138ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
45232025-03-21T02:08:27.164ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
45242025-03-21T02:08:27.164ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
45252025-03-21T02:08:27.164ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
45262025-03-21T02:08:27.164ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "fffa6b7b-37d4-4bee-ba08-c705cc05a821", "content-length": "372", "date": "Fri, 21 Mar 2025 02:08:26 GMT"} })
45272025-03-21T02:08:27.165ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45282025-03-21T02:08:27.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45292025-03-21T02:08:27.166ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45302025-03-21T02:08:27.167ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45312025-03-21T02:08:27.167ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45322025-03-21T02:08:27.169ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
45332025-03-21T02:08:27.169ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
45342025-03-21T02:08:27.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
45352025-03-21T02:08:27.169ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
45362025-03-21T02:08:27.169ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
45372025-03-21T02:08:27.170ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
45382025-03-21T02:08:27.170ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
45392025-03-21T02:08:27.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
45402025-03-21T02:08:27.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
45412025-03-21T02:08:27.172ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45422025-03-21T02:08:27.173ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
45432025-03-21T02:08:27.173ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 254.924918ms iteration = 9 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
45442025-03-21T02:08:27.175ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
45452025-03-21T02:08:27.175ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
45462025-03-21T02:08:27.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
45472025-03-21T02:08:27.203ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45482025-03-21T02:08:27.204ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
45492025-03-21T02:08:27.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
45502025-03-21T02:08:27.206ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
45512025-03-21T02:08:27.206ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45522025-03-21T02:08:27.206ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45532025-03-21T02:08:27.206ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
45542025-03-21T02:08:27.207ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
45552025-03-21T02:08:27.207ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45562025-03-21T02:08:27.208ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
45572025-03-21T02:08:27.208ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
45582025-03-21T02:08:27.208ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
45592025-03-21T02:08:27.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
45602025-03-21T02:08:27.210ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
45612025-03-21T02:08:27.210ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45622025-03-21T02:08:27.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45632025-03-21T02:08:27.212ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
45642025-03-21T02:08:27.212ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
45652025-03-21T02:08:27.212ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 294.228092ms iteration = 9 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
45662025-03-21T02:08:27.221ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
45672025-03-21T02:08:27.221ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
45682025-03-21T02:08:27.221ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
45692025-03-21T02:08:27.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
45702025-03-21T02:08:27.223ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45712025-03-21T02:08:27.225ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
45722025-03-21T02:08:27.225ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
45732025-03-21T02:08:27.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
45742025-03-21T02:08:27.226ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
45752025-03-21T02:08:27.226ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
45762025-03-21T02:08:27.226ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e08d2200-f303-43ef-9097-29ccb0a6cb41", "content-length": "26", "date": "Fri, 21 Mar 2025 02:08:26 GMT"} })
45772025-03-21T02:08:27.226ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
45782025-03-21T02:08:27.228ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
45792025-03-21T02:08:27.228ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
45802025-03-21T02:08:27.228ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
45812025-03-21T02:08:27.228ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-21 02:04:30.886360 UTC dns_group = internal generation = 1
45822025-03-21T02:08:27.228ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 310.447474ms iteration = 5 status = {"generation":1}
45832025-03-21T02:08:27.229ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) }
45842025-03-21T02:08:27.229ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 311.148487ms iteration = 5 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
45852025-03-21T02:08:27.235ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
45862025-03-21T02:08:27.235ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
45872025-03-21T02:08:27.237ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
45882025-03-21T02:08:27.250ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45892025-03-21T02:08:27.254ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
45902025-03-21T02:08:27.254ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
45912025-03-21T02:08:27.254ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45922025-03-21T02:08:27.254ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
45932025-03-21T02:08:27.257ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
45942025-03-21T02:08:27.257ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
45952025-03-21T02:08:27.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
45962025-03-21T02:08:27.257ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 338.813329ms iteration = 9 status = 0
45972025-03-21T02:08:27.258ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
45982025-03-21T02:08:27.258ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
45992025-03-21T02:08:27.258ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-21 02:04:31.662541 UTC dns_group = external generation = 2
46002025-03-21T02:08:27.258ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 340.07833ms iteration = 5 status = {"generation":2}
46012025-03-21T02:08:27.295ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
46022025-03-21T02:08:27.295ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 376.881301ms iteration = 5 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
46032025-03-21T02:08:27.331ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "9259bb6c-d128-459a-8262-b3236f7b555f", "date": "Fri, 21 Mar 2025 02:08:27 GMT"} })
46042025-03-21T02:08:27.331ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46052025-03-21T02:08:27.332ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46062025-03-21T02:08:27.334ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46072025-03-21T02:08:27.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46082025-03-21T02:08:27.343ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
46092025-03-21T02:08:27.343ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "92a9e386-2c4e-4c5b-87eb-76fac95df255", "content-length": "1", "date": "Fri, 21 Mar 2025 02:08:27 GMT"} })
46102025-03-21T02:08:27.345ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
46112025-03-21T02:08:27.345ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
46122025-03-21T02:08:27.355ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
46132025-03-21T02:08:27.355ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "876fe4a6-1f5a-45b1-b292-8c155d7323b6", "date": "Fri, 21 Mar 2025 02:08:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
46142025-03-21T02:08:27.355ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
46152025-03-21T02:08:27.357ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "cffff332-85a1-4974-97a5-93e514a35b68", "date": "Fri, 21 Mar 2025 02:08:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
46162025-03-21T02:08:27.357ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
46172025-03-21T02:08:27.359ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
46182025-03-21T02:08:27.359ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 440.951576ms iteration = 7 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
46192025-03-21T02:08:27.380ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 462.228058ms iteration = 9 status = {}
46202025-03-21T02:08:27.383ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
46212025-03-21T02:08:27.384ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "886bd15b-033e-4fa8-ac24-d3a076171174", "date": "Fri, 21 Mar 2025 02:08:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
46222025-03-21T02:08:27.384ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
46232025-03-21T02:08:27.386ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "16065c43-fb83-4559-9a1a-d1d3df10c346", "date": "Fri, 21 Mar 2025 02:08:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
46242025-03-21T02:08:27.386ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
46252025-03-21T02:08:27.387ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "d877259f-c097-4cf4-af4f-a7ed65db924f", "date": "Fri, 21 Mar 2025 02:08:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
46262025-03-21T02:08:27.387ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
46272025-03-21T02:08:27.389ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
46282025-03-21T02:08:27.389ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 471.044921ms iteration = 7 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
46292025-03-21T02:08:27.402ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
46302025-03-21T02:08:27.404ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c002ef77-155e-4991-8093-28513a48e626", "content-length": "691", "date": "Fri, 21 Mar 2025 02:08:27 GMT"} })
46312025-03-21T02:08:27.404ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
46322025-03-21T02:08:27.404ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
46332025-03-21T02:08:27.404ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
46342025-03-21T02:08:27.405ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 487.407226ms iteration = 9 status = null
46352025-03-21T02:08:27.406ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46362025-03-21T02:08:27.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46372025-03-21T02:08:27.407ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 488.429293ms iteration = 9 status = {}
46382025-03-21T02:08:27.410ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
46392025-03-21T02:08:27.410ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
46402025-03-21T02:08:27.410ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "fafdbeda-ef71-41eb-bf2d-889fac112ae2", "date": "Fri, 21 Mar 2025 02:08:27 GMT"} })
46412025-03-21T02:08:27.411ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 493.212206ms iteration = 9 status = {}
46422025-03-21T02:08:27.448ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46432025-03-21T02:08:27.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46442025-03-21T02:08:27.450ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
46452025-03-21T02:08:27.450ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46462025-03-21T02:08:27.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46472025-03-21T02:08:27.461ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
46482025-03-21T02:08:27.461ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "985e5940-a86d-4089-9cbc-9b0fd114695c", "content-length": "134", "date": "Fri, 21 Mar 2025 02:08:27 GMT"} })
46492025-03-21T02:08:27.461ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
46502025-03-21T02:08:27.461ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
46512025-03-21T02:08:27.461ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
46522025-03-21T02:08:27.461ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
46532025-03-21T02:08:27.461ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
46542025-03-21T02:08:27.461ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
46552025-03-21T02:08:27.461ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
46562025-03-21T02:08:27.461ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
46572025-03-21T02:08:27.461ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
46582025-03-21T02:08:27.461ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
46592025-03-21T02:08:27.462ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
46602025-03-21T02:08:27.462ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46612025-03-21T02:08:27.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46622025-03-21T02:08:27.464ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
46632025-03-21T02:08:27.464ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "75fba2a0-5bb9-463f-ac63-4b70d6aa17aa", "content-length": "2", "date": "Fri, 21 Mar 2025 02:08:27 GMT"} })
46642025-03-21T02:08:27.464ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
46652025-03-21T02:08:27.464ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0b9447f-22f8-452a-968e-19432d1e5be7", "content-length": "52", "date": "Fri, 21 Mar 2025 02:08:27 GMT"} })
46662025-03-21T02:08:27.464ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
46672025-03-21T02:08:27.464ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
46682025-03-21T02:08:27.556ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "53e6d01b-4043-491a-9092-2832f05204c8", "date": "Fri, 21 Mar 2025 02:08:26 GMT"} })
46692025-03-21T02:08:27.556ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
46702025-03-21T02:08:27.556ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
46712025-03-21T02:08:27.556ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "76493206-ec4f-4aba-a2a7-c8b4efbe35dd", "content-length": "682", "date": "Fri, 21 Mar 2025 02:08:26 GMT"} })
46722025-03-21T02:08:27.556ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46732025-03-21T02:08:27.556ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46742025-03-21T02:08:27.560ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46752025-03-21T02:08:27.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46762025-03-21T02:08:27.563ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46772025-03-21T02:08:27.564ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46782025-03-21T02:08:27.566ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46792025-03-21T02:08:27.566ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46802025-03-21T02:08:27.573ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 654.84238ms iteration = 9 status = {}
46812025-03-21T02:08:29.718ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 248 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 733b2b41-71e3-4a32-be72-e9ed68e6b75f response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
46822025-03-21T02:08:36.918ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 26 reason = Timeout
46832025-03-21T02:08:36.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 26 reason = Timeout
46842025-03-21T02:08:36.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
46852025-03-21T02:08:36.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46862025-03-21T02:08:36.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46872025-03-21T02:08:36.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
46882025-03-21T02:08:36.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46892025-03-21T02:08:36.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
46902025-03-21T02:08:36.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
46912025-03-21T02:08:36.923ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
46922025-03-21T02:08:36.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46932025-03-21T02:08:36.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46942025-03-21T02:08:36.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46952025-03-21T02:08:36.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46962025-03-21T02:08:36.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46972025-03-21T02:08:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
46982025-03-21T02:08:36.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46992025-03-21T02:08:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47002025-03-21T02:08:36.937ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
47012025-03-21T02:08:36.937ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.98351ms iteration = 26 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
47022025-03-21T02:08:36.953ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.428779ms iteration = 26 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
47032025-03-21T02:08:39.719ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 264 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = db9ba9e4-a718-45c1-81dc-6c81828a1e9d response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
47042025-03-21T02:08:46.918ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 27 reason = Timeout
47052025-03-21T02:08:46.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47062025-03-21T02:08:46.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
47072025-03-21T02:08:46.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47082025-03-21T02:08:46.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47092025-03-21T02:08:46.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47102025-03-21T02:08:46.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47112025-03-21T02:08:46.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47122025-03-21T02:08:46.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47132025-03-21T02:08:46.923ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
47142025-03-21T02:08:46.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47152025-03-21T02:08:46.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47162025-03-21T02:08:46.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47172025-03-21T02:08:46.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47182025-03-21T02:08:46.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47192025-03-21T02:08:46.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
47202025-03-21T02:08:46.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47212025-03-21T02:08:46.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47222025-03-21T02:08:46.938ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
47232025-03-21T02:08:46.938ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.495567ms iteration = 27 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
47242025-03-21T02:08:47.050ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 132.281843ms iteration = 27 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
47252025-03-21T02:08:49.718ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 269 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 0c5e97dd-49e1-4e22-9c91-c124ce19478b response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
47262025-03-21T02:08:54.710ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = b2304f9d-ff36-4ee3-8ef3-3effe6dc03af roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
47272025-03-21T02:08:54.710ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = b2304f9d-ff36-4ee3-8ef3-3effe6dc03af resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
47282025-03-21T02:08:54.713ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3720 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = b2304f9d-ff36-4ee3-8ef3-3effe6dc03af response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
47292025-03-21T02:08:54.714ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = cceffc0a-c0f6-4f39-8d9d-85390fd33381 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
47302025-03-21T02:08:54.714ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = cceffc0a-c0f6-4f39-8d9d-85390fd33381 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
47312025-03-21T02:08:54.716ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2479 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36229 req_id = cceffc0a-c0f6-4f39-8d9d-85390fd33381 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
47322025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 28 reason = Timeout
47332025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47342025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
47352025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 10 reason = Timeout
47362025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
47372025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
47382025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
47392025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
47402025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
47412025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47422025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
47432025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 10 reason = Timeout
47442025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
47452025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
47462025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
47472025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 10 reason = Timeout
47482025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 10 reason = Timeout
47492025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
47502025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
47512025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 10 reason = Timeout
47522025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
47532025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 10 reason = Timeout
47542025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
47552025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
47562025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 10 reason = Timeout
47572025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
47582025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 10 reason = Timeout
47592025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 11 reason = Timeout
47602025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
47612025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
47622025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
47632025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
47642025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 10 reason = Timeout
47652025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
47662025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 10 reason = Timeout
47672025-03-21T02:08:56.918ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
47682025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
47692025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 10 reason = Timeout
47702025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47712025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47722025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
47732025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47742025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47752025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
47762025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
47772025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
47782025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
47792025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
47802025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
47812025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
47822025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
47832025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
47842025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
47852025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
47862025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
47872025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
47882025-03-21T02:08:56.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47892025-03-21T02:08:56.920ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47902025-03-21T02:08:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
47912025-03-21T02:08:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47922025-03-21T02:08:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
47932025-03-21T02:08:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47942025-03-21T02:08:56.921ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.46375ms iteration = 10 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
47952025-03-21T02:08:56.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47962025-03-21T02:08:56.922ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1245, tv_nsec: 546373451 } })
47972025-03-21T02:08:56.923ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
47982025-03-21T02:08:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
47992025-03-21T02:08:56.923ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48002025-03-21T02:08:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48012025-03-21T02:08:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
48022025-03-21T02:08:56.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48032025-03-21T02:08:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48042025-03-21T02:08:56.925ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48052025-03-21T02:08:56.925ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48062025-03-21T02:08:56.926ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1245, tv_nsec: 550277949 } })
48072025-03-21T02:08:56.926ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
48082025-03-21T02:08:56.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
48092025-03-21T02:08:56.927ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
48102025-03-21T02:08:56.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
48112025-03-21T02:08:56.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
48122025-03-21T02:08:56.927ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48132025-03-21T02:08:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
48142025-03-21T02:08:56.930ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
48152025-03-21T02:08:56.930ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
48162025-03-21T02:08:56.930ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
48172025-03-21T02:08:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
48182025-03-21T02:08:56.930ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48192025-03-21T02:08:56.932ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
48202025-03-21T02:08:56.932ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48212025-03-21T02:08:56.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48222025-03-21T02:08:56.933ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48232025-03-21T02:08:56.933ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 15.013521ms iteration = 10 status = {"errors":[],"requests_created_ok":[]}
48242025-03-21T02:08:56.933ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
48252025-03-21T02:08:56.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
48262025-03-21T02:08:56.934ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
48272025-03-21T02:08:56.934ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48282025-03-21T02:08:56.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
48292025-03-21T02:08:56.935ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
48302025-03-21T02:08:56.935ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48312025-03-21T02:08:56.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48322025-03-21T02:08:56.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48332025-03-21T02:08:56.937ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48342025-03-21T02:08:56.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48352025-03-21T02:08:56.938ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48362025-03-21T02:08:56.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
48372025-03-21T02:08:56.938ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48382025-03-21T02:08:56.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48392025-03-21T02:08:56.940ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
48402025-03-21T02:08:56.940ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48412025-03-21T02:08:56.940ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48422025-03-21T02:08:56.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48432025-03-21T02:08:56.940ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1245, tv_nsec: 564433175 } })
48442025-03-21T02:08:56.941ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
48452025-03-21T02:08:56.942ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
48462025-03-21T02:08:56.942ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48472025-03-21T02:08:56.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
48482025-03-21T02:08:56.943ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
48492025-03-21T02:08:56.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48502025-03-21T02:08:56.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48512025-03-21T02:08:56.959ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
48522025-03-21T02:08:56.959ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 40.846321ms iteration = 28 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
48532025-03-21T02:08:56.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
48542025-03-21T02:08:56.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
48552025-03-21T02:08:56.966ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 48.359272ms iteration = 28 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
48562025-03-21T02:08:56.967ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 48.740068ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
48572025-03-21T02:08:56.970ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 52.331211ms iteration = 10 status = {"errors":[],"finish_invoked_ok":[]}
48582025-03-21T02:08:56.972ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 54.299745ms iteration = 10 status = {"errors":[],"garbage_collect_requested":[]}
48592025-03-21T02:08:56.973ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48602025-03-21T02:08:56.973ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
48612025-03-21T02:08:56.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
48622025-03-21T02:08:56.974ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 55.750311ms iteration = 10 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
48632025-03-21T02:08:56.975ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48642025-03-21T02:08:56.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
48652025-03-21T02:08:56.975ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
48662025-03-21T02:08:56.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
48672025-03-21T02:08:56.975ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
48682025-03-21T02:08:56.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
48692025-03-21T02:08:56.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
48702025-03-21T02:08:56.977ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
48712025-03-21T02:08:56.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
48722025-03-21T02:08:56.978ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48732025-03-21T02:08:56.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48742025-03-21T02:08:56.979ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
48752025-03-21T02:08:56.979ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 61.140534ms iteration = 10 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
48762025-03-21T02:08:56.979ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48772025-03-21T02:08:56.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48782025-03-21T02:08:56.981ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48792025-03-21T02:08:56.982ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
48802025-03-21T02:08:56.982ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 64.29732ms iteration = 10 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
48812025-03-21T02:08:56.983ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
48822025-03-21T02:08:56.983ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
48832025-03-21T02:08:56.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
48842025-03-21T02:08:56.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
48852025-03-21T02:08:56.984ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
48862025-03-21T02:08:56.984ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48872025-03-21T02:08:56.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48882025-03-21T02:08:56.985ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 66.730281ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
48892025-03-21T02:08:56.986ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 68.023845ms iteration = 10 status = 0
48902025-03-21T02:08:56.991ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
48912025-03-21T02:08:56.991ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
48922025-03-21T02:08:56.991ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "771fa1ee-30d2-4019-beb4-e4242bd5345d", "content-length": "26", "date": "Fri, 21 Mar 2025 02:08:56 GMT"} })
48932025-03-21T02:08:56.991ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
48942025-03-21T02:08:56.992ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
48952025-03-21T02:08:56.993ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 74.718581ms iteration = 11 status = {"physical_disks_added":0}
48962025-03-21T02:08:57.014ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
48972025-03-21T02:08:57.014ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
48982025-03-21T02:08:57.014ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "09a3acf7-68f6-4951-9d01-1a064335d617", "content-length": "26", "date": "Fri, 21 Mar 2025 02:08:56 GMT"} })
48992025-03-21T02:08:57.014ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
49002025-03-21T02:08:57.016ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
49012025-03-21T02:08:57.016ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
49022025-03-21T02:08:57.016ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
49032025-03-21T02:08:57.016ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 98.558416ms iteration = 10 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
49042025-03-21T02:08:57.026ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
49052025-03-21T02:08:57.026ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
49062025-03-21T02:08:57.026ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "07742ddd-676a-43c8-885a-ef04f2331baa", "content-length": "26", "date": "Fri, 21 Mar 2025 02:08:57 GMT"} })
49072025-03-21T02:08:57.026ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
49082025-03-21T02:08:57.028ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
49092025-03-21T02:08:57.074ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
49102025-03-21T02:08:57.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
49112025-03-21T02:08:57.076ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49122025-03-21T02:08:57.078ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
49132025-03-21T02:08:57.078ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
49142025-03-21T02:08:57.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
49152025-03-21T02:08:57.079ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49162025-03-21T02:08:57.081ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
49172025-03-21T02:08:57.081ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
49182025-03-21T02:08:57.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
49192025-03-21T02:08:57.116ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 197.918005ms iteration = 10 status = {}
49202025-03-21T02:08:57.233ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
49212025-03-21T02:08:57.233ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "3216675b-3607-4360-acbb-aa07f1e8f8b7", "content-length": "1", "date": "Fri, 21 Mar 2025 02:08:56 GMT"} })
49222025-03-21T02:08:57.234ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
49232025-03-21T02:08:57.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
49242025-03-21T02:08:57.257ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
49252025-03-21T02:08:57.257ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
49262025-03-21T02:08:57.258ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "656dab41-0180-4922-aa24-a7057e04919b", "content-length": "372", "date": "Fri, 21 Mar 2025 02:08:57 GMT"} })
49272025-03-21T02:08:57.258ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49282025-03-21T02:08:57.258ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49292025-03-21T02:08:57.266ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
49302025-03-21T02:08:57.267ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "9662599e-4701-4a27-9aa2-709263c33f2c", "content-length": "691", "date": "Fri, 21 Mar 2025 02:08:57 GMT"} })
49312025-03-21T02:08:57.267ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
49322025-03-21T02:08:57.267ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
49332025-03-21T02:08:57.267ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
49342025-03-21T02:08:57.268ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 349.908122ms iteration = 10 status = null
49352025-03-21T02:08:57.268ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 350.206326ms iteration = 10 status = {}
49362025-03-21T02:08:57.307ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
49372025-03-21T02:08:57.307ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
49382025-03-21T02:08:57.308ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "69adb587-a07f-42f0-a2d6-cd37f98e5a05", "date": "Fri, 21 Mar 2025 02:08:57 GMT"} })
49392025-03-21T02:08:57.308ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49402025-03-21T02:08:57.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49412025-03-21T02:08:57.309ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49422025-03-21T02:08:57.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49432025-03-21T02:08:57.313ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49442025-03-21T02:08:57.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49452025-03-21T02:08:57.316ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
49462025-03-21T02:08:57.316ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
49472025-03-21T02:08:57.316ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "bd7f4a4e-d7d0-414b-9755-c44d2a1d13d1", "date": "Fri, 21 Mar 2025 02:08:57 GMT"} })
49482025-03-21T02:08:57.318ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 400.181196ms iteration = 10 status = {}
49492025-03-21T02:08:57.325ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49502025-03-21T02:08:57.325ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49512025-03-21T02:08:57.327ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
49522025-03-21T02:08:57.327ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49532025-03-21T02:08:57.328ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49542025-03-21T02:08:57.337ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
49552025-03-21T02:08:57.338ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "106bec67-148b-4d95-9402-7970eebce2d8", "content-length": "134", "date": "Fri, 21 Mar 2025 02:08:56 GMT"} })
49562025-03-21T02:08:57.338ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
49572025-03-21T02:08:57.338ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
49582025-03-21T02:08:57.338ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
49592025-03-21T02:08:57.338ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
49602025-03-21T02:08:57.338ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
49612025-03-21T02:08:57.338ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
49622025-03-21T02:08:57.338ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
49632025-03-21T02:08:57.338ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
49642025-03-21T02:08:57.338ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
49652025-03-21T02:08:57.338ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
49662025-03-21T02:08:57.338ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
49672025-03-21T02:08:57.338ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49682025-03-21T02:08:57.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49692025-03-21T02:08:57.341ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
49702025-03-21T02:08:57.341ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c633f91-0e90-4a8d-9844-c834fe918f5e", "content-length": "2", "date": "Fri, 21 Mar 2025 02:08:56 GMT"} })
49712025-03-21T02:08:57.341ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
49722025-03-21T02:08:57.341ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fc5e207-77fb-45d8-8af4-c975dcc8942a", "content-length": "52", "date": "Fri, 21 Mar 2025 02:08:56 GMT"} })
49732025-03-21T02:08:57.341ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
49742025-03-21T02:08:57.341ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
49752025-03-21T02:08:57.437ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "9d0078fa-fb3d-4343-a15b-8b5c03767e77", "date": "Fri, 21 Mar 2025 02:08:57 GMT"} })
49762025-03-21T02:08:57.437ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
49772025-03-21T02:08:57.437ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
49782025-03-21T02:08:57.437ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "35b14261-d54b-4c2b-a4c5-3e7dba48a0d4", "content-length": "682", "date": "Fri, 21 Mar 2025 02:08:57 GMT"} })
49792025-03-21T02:08:57.437ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49802025-03-21T02:08:57.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49812025-03-21T02:08:57.441ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49822025-03-21T02:08:57.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49832025-03-21T02:08:57.445ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49842025-03-21T02:08:57.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49852025-03-21T02:08:57.447ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49862025-03-21T02:08:57.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49872025-03-21T02:08:57.454ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 535.751007ms iteration = 10 status = {}
49882025-03-21T02:08:59.719ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 318 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = a38c5a21-541c-48ea-aafe-840025b4d5f2 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
49892025-03-21T02:09:06.918ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 29 reason = Timeout
49902025-03-21T02:09:06.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49912025-03-21T02:09:06.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
49922025-03-21T02:09:06.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49932025-03-21T02:09:06.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49942025-03-21T02:09:06.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49952025-03-21T02:09:06.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49962025-03-21T02:09:06.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49972025-03-21T02:09:06.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49982025-03-21T02:09:06.923ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
49992025-03-21T02:09:06.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50002025-03-21T02:09:06.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50012025-03-21T02:09:06.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50022025-03-21T02:09:06.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50032025-03-21T02:09:06.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50042025-03-21T02:09:06.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
50052025-03-21T02:09:06.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50062025-03-21T02:09:06.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50072025-03-21T02:09:06.937ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
50082025-03-21T02:09:06.937ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.791248ms iteration = 29 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
50092025-03-21T02:09:06.958ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 39.257803ms iteration = 29 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
50102025-03-21T02:09:09.718ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 264 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 525dd81b-3590-4734-8122-ef3aff4ff98d response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
50112025-03-21T02:09:16.919ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 30 reason = Timeout
50122025-03-21T02:09:16.919ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
50132025-03-21T02:09:16.919ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
50142025-03-21T02:09:16.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50152025-03-21T02:09:16.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
50162025-03-21T02:09:16.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50172025-03-21T02:09:16.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
50182025-03-21T02:09:16.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
50192025-03-21T02:09:16.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50202025-03-21T02:09:16.923ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
50212025-03-21T02:09:16.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50222025-03-21T02:09:16.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50232025-03-21T02:09:16.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50242025-03-21T02:09:16.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50252025-03-21T02:09:16.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50262025-03-21T02:09:16.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
50272025-03-21T02:09:16.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50282025-03-21T02:09:16.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50292025-03-21T02:09:16.936ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
50302025-03-21T02:09:16.936ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.080257ms iteration = 30 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
50312025-03-21T02:09:16.949ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 30.570272ms iteration = 30 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
50322025-03-21T02:09:19.719ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 294 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 82721ecc-698d-464b-a6ae-48483ff2b45c response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
50332025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 11 reason = Timeout
50342025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
50352025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
50362025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
50372025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 2 reason = Timeout
50382025-03-21T02:09:26.918ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
50392025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50402025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 11 reason = Timeout
50412025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 6 reason = Timeout
50422025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
50432025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
50442025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 6 reason = Timeout
50452025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
50462025-03-21T02:09:26.918ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
50472025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 11 reason = Timeout
50482025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
50492025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50502025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
50512025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
50522025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
50532025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50542025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
50552025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 6 reason = Timeout
50562025-03-21T02:09:26.918ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 01:59:26.918729378 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
50572025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 11 reason = Timeout
50582025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
50592025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
50602025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
50612025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
50622025-03-21T02:09:26.918ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
50632025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 15.44\u{b5}s iteration = 7 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
50642025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 11 reason = Timeout
50652025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
50662025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
50672025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
50682025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 11 reason = Timeout
50692025-03-21T02:09:26.918ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
50702025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
50712025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50722025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
50732025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
50742025-03-21T02:09:26.918ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
50752025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 6 reason = Timeout
50762025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
50772025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 11 reason = Timeout
50782025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
50792025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
50802025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
50812025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
50822025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 11 reason = Timeout
50832025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
50842025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
50852025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
50862025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
50872025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
50882025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
50892025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 11 reason = Timeout
50902025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50912025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 31 reason = Timeout
50922025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 2 reason = Timeout
50932025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
50942025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
50952025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 6 reason = Timeout
50962025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
50972025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 11 reason = Timeout
50982025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
50992025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
51002025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 11 reason = Timeout
51012025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 12 reason = Timeout
51022025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
51032025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 3 reason = Timeout
51042025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
51052025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51062025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
51072025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
51082025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51092025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
51102025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
51112025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
51122025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51132025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
51142025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51152025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
51162025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
51172025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
51182025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
51192025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
51202025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
51212025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51222025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
51232025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
51242025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
51252025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
51262025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51272025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
51282025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51292025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
51302025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
51312025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
51322025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
51332025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
51342025-03-21T02:09:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51352025-03-21T02:09:26.919ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1275, tv_nsec: 542938782 } })
51362025-03-21T02:09:26.919ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1275, tv_nsec: 542950632 } })
51372025-03-21T02:09:26.920ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
51382025-03-21T02:09:26.920ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.406005ms iteration = 6 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
51392025-03-21T02:09:26.920ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
51402025-03-21T02:09:26.920ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.448795ms iteration = 6 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
51412025-03-21T02:09:26.921ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1275, tv_nsec: 544744123 } })
51422025-03-21T02:09:26.921ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
51432025-03-21T02:09:26.922ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.824447ms iteration = 11 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
51442025-03-21T02:09:26.922ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51452025-03-21T02:09:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51462025-03-21T02:09:26.923ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 4.792354ms iteration = 11 status = {"errors":[],"finish_invoked_ok":[]}
51472025-03-21T02:09:26.923ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 01:59:26.918729378 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
51482025-03-21T02:09:26.923ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.18032ms iteration = 6 status = {"expiration":"2025-03-21T01:59:26.918729378Z","pruned":[]}
51492025-03-21T02:09:26.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51502025-03-21T02:09:26.924ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 6.2974ms iteration = 11 status = {"errors":[],"requests_created_ok":[]}
51512025-03-21T02:09:26.925ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51522025-03-21T02:09:26.926ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51532025-03-21T02:09:26.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
51542025-03-21T02:09:26.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51552025-03-21T02:09:26.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51562025-03-21T02:09:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
51572025-03-21T02:09:26.928ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51582025-03-21T02:09:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51592025-03-21T02:09:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
51602025-03-21T02:09:26.929ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
51612025-03-21T02:09:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
51622025-03-21T02:09:26.931ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51632025-03-21T02:09:26.931ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
51642025-03-21T02:09:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
51652025-03-21T02:09:26.931ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 12.862494ms iteration = 6 status = {"errors":[],"found_port_ok":[]}
51662025-03-21T02:09:26.932ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51672025-03-21T02:09:26.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
51682025-03-21T02:09:26.933ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
51692025-03-21T02:09:26.933ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51702025-03-21T02:09:26.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
51712025-03-21T02:09:26.934ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
51722025-03-21T02:09:26.934ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
51732025-03-21T02:09:26.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
51742025-03-21T02:09:26.935ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
51752025-03-21T02:09:26.935ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51762025-03-21T02:09:26.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51772025-03-21T02:09:26.941ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51782025-03-21T02:09:26.942ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51792025-03-21T02:09:26.943ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1275, tv_nsec: 566994121 } })
51802025-03-21T02:09:26.944ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
51812025-03-21T02:09:26.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
51822025-03-21T02:09:26.944ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51832025-03-21T02:09:26.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51842025-03-21T02:09:26.944ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
51852025-03-21T02:09:26.944ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
51862025-03-21T02:09:26.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
51872025-03-21T02:09:26.945ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
51882025-03-21T02:09:26.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
51892025-03-21T02:09:26.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51902025-03-21T02:09:26.949ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51912025-03-21T02:09:26.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51922025-03-21T02:09:26.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51932025-03-21T02:09:26.950ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51942025-03-21T02:09:26.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51952025-03-21T02:09:26.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51962025-03-21T02:09:26.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
51972025-03-21T02:09:26.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51982025-03-21T02:09:26.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51992025-03-21T02:09:26.953ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52002025-03-21T02:09:26.954ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
52012025-03-21T02:09:26.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
52022025-03-21T02:09:26.955ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
52032025-03-21T02:09:26.955ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52042025-03-21T02:09:26.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52052025-03-21T02:09:26.957ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52062025-03-21T02:09:26.958ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1275, tv_nsec: 581963381 } })
52072025-03-21T02:09:26.959ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
52082025-03-21T02:09:26.961ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
52092025-03-21T02:09:26.961ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
52102025-03-21T02:09:26.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
52112025-03-21T02:09:26.964ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52122025-03-21T02:09:26.966ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
52132025-03-21T02:09:26.966ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
52142025-03-21T02:09:26.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
52152025-03-21T02:09:26.966ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 48.081317ms iteration = 11 status = {"errors":[],"garbage_collect_requested":[]}
52162025-03-21T02:09:26.966ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52172025-03-21T02:09:26.968ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
52182025-03-21T02:09:26.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52192025-03-21T02:09:26.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
52202025-03-21T02:09:26.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
52212025-03-21T02:09:26.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
52222025-03-21T02:09:26.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
52232025-03-21T02:09:26.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
52242025-03-21T02:09:26.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52252025-03-21T02:09:26.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52262025-03-21T02:09:26.986ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
52272025-03-21T02:09:26.986ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
52282025-03-21T02:09:26.986ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 67.553716ms iteration = 6 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
52292025-03-21T02:09:26.993ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
52302025-03-21T02:09:26.993ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
52312025-03-21T02:09:26.993ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
52322025-03-21T02:09:26.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
52332025-03-21T02:09:26.995ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52342025-03-21T02:09:26.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
52352025-03-21T02:09:26.997ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
52362025-03-21T02:09:26.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
52372025-03-21T02:09:26.999ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
52382025-03-21T02:09:26.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
52392025-03-21T02:09:27.000ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52402025-03-21T02:09:27.000ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
52412025-03-21T02:09:27.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
52422025-03-21T02:09:27.002ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
52432025-03-21T02:09:27.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
52442025-03-21T02:09:27.002ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52452025-03-21T02:09:27.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52462025-03-21T02:09:27.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
52472025-03-21T02:09:27.003ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
52482025-03-21T02:09:27.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
52492025-03-21T02:09:27.005ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
52502025-03-21T02:09:27.005ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
52512025-03-21T02:09:27.005ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 87.039265ms iteration = 6 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
52522025-03-21T02:09:27.006ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52532025-03-21T02:09:27.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52542025-03-21T02:09:27.007ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
52552025-03-21T02:09:27.007ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
52562025-03-21T02:09:27.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
52572025-03-21T02:09:27.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
52582025-03-21T02:09:27.008ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
52592025-03-21T02:09:27.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
52602025-03-21T02:09:27.009ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52612025-03-21T02:09:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52622025-03-21T02:09:27.010ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
52632025-03-21T02:09:27.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
52642025-03-21T02:09:27.012ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52652025-03-21T02:09:27.013ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
52662025-03-21T02:09:27.013ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
52672025-03-21T02:09:27.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
52682025-03-21T02:09:27.015ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
52692025-03-21T02:09:27.015ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 96.356818ms iteration = 31 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
52702025-03-21T02:09:27.016ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52712025-03-21T02:09:27.017ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52722025-03-21T02:09:27.017ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52732025-03-21T02:09:27.019ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
52742025-03-21T02:09:27.019ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
52752025-03-21T02:09:27.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
52762025-03-21T02:09:27.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
52772025-03-21T02:09:27.021ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52782025-03-21T02:09:27.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52792025-03-21T02:09:27.022ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
52802025-03-21T02:09:27.024ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 105.567438ms iteration = 11 status = 0
52812025-03-21T02:09:27.024ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
52822025-03-21T02:09:27.025ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
52832025-03-21T02:09:27.025ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
52842025-03-21T02:09:27.025ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
52852025-03-21T02:09:27.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
52862025-03-21T02:09:27.027ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52872025-03-21T02:09:27.028ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
52882025-03-21T02:09:27.028ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
52892025-03-21T02:09:27.028ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
52902025-03-21T02:09:27.028ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
52912025-03-21T02:09:27.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
52922025-03-21T02:09:27.031ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
52932025-03-21T02:09:27.031ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
52942025-03-21T02:09:27.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
52952025-03-21T02:09:27.032ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
52962025-03-21T02:09:27.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
52972025-03-21T02:09:27.033ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52982025-03-21T02:09:27.034ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
52992025-03-21T02:09:27.034ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
53002025-03-21T02:09:27.034ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
53012025-03-21T02:09:27.034ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53022025-03-21T02:09:27.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
53032025-03-21T02:09:27.037ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
53042025-03-21T02:09:27.037ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
53052025-03-21T02:09:27.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
53062025-03-21T02:09:27.038ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
53072025-03-21T02:09:27.038ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
53082025-03-21T02:09:27.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
53092025-03-21T02:09:27.045ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
53102025-03-21T02:09:27.046ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
53112025-03-21T02:09:27.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
53122025-03-21T02:09:27.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
53132025-03-21T02:09:27.051ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53142025-03-21T02:09:27.056ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
53152025-03-21T02:09:27.056ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53162025-03-21T02:09:27.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53172025-03-21T02:09:27.056ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 138.056733ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
53182025-03-21T02:09:27.057ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 138.352279ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
53192025-03-21T02:09:27.059ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
53202025-03-21T02:09:27.059ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
53212025-03-21T02:09:27.059ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
53222025-03-21T02:09:27.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
53232025-03-21T02:09:27.059ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 140.890843ms iteration = 31 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
53242025-03-21T02:09:27.062ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
53252025-03-21T02:09:27.062ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
53262025-03-21T02:09:27.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
53272025-03-21T02:09:27.071ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
53282025-03-21T02:09:27.071ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 153.101876ms iteration = 11 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
53292025-03-21T02:09:27.071ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
53302025-03-21T02:09:27.071ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 153.001494ms iteration = 11 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
53312025-03-21T02:09:27.074ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 155.776932ms iteration = 12 status = {"physical_disks_added":0}
53322025-03-21T02:09:27.084ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
53332025-03-21T02:09:27.084ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
53342025-03-21T02:09:27.084ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
53352025-03-21T02:09:27.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
53362025-03-21T02:09:27.085ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53372025-03-21T02:09:27.088ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
53382025-03-21T02:09:27.088ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
53392025-03-21T02:09:27.089ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
53402025-03-21T02:09:27.091ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) }
53412025-03-21T02:09:27.091ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 172.93287ms iteration = 6 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
53422025-03-21T02:09:27.093ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53432025-03-21T02:09:27.096ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
53442025-03-21T02:09:27.096ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
53452025-03-21T02:09:27.096ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53462025-03-21T02:09:27.096ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "82d6202a-8590-488d-85d4-436cbdacadc1", "content-length": "26", "date": "Fri, 21 Mar 2025 02:09:26 GMT"} })
53472025-03-21T02:09:27.096ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
53482025-03-21T02:09:27.097ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
53492025-03-21T02:09:27.097ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
53502025-03-21T02:09:27.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
53512025-03-21T02:09:27.098ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
53522025-03-21T02:09:27.098ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
53532025-03-21T02:09:27.098ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
53542025-03-21T02:09:27.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
53552025-03-21T02:09:27.099ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 179.958723ms iteration = 11 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
53562025-03-21T02:09:27.100ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
53572025-03-21T02:09:27.100ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
53582025-03-21T02:09:27.100ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-21 02:04:31.662541 UTC dns_group = external generation = 2
53592025-03-21T02:09:27.100ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 181.234025ms iteration = 6 status = {"generation":2}
53602025-03-21T02:09:27.102ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53612025-03-21T02:09:27.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53622025-03-21T02:09:27.114ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53632025-03-21T02:09:27.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53642025-03-21T02:09:27.114ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
53652025-03-21T02:09:27.114ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
53662025-03-21T02:09:27.115ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "788755a4-dd65-49a7-a803-62020e3c2c86", "content-length": "26", "date": "Fri, 21 Mar 2025 02:09:26 GMT"} })
53672025-03-21T02:09:27.115ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
53682025-03-21T02:09:27.118ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
53692025-03-21T02:09:27.120ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
53702025-03-21T02:09:27.120ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
53712025-03-21T02:09:27.120ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 201.455197ms iteration = 11 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
53722025-03-21T02:09:27.184ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
53732025-03-21T02:09:27.184ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 265.396631ms iteration = 6 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
53742025-03-21T02:09:27.184ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
53752025-03-21T02:09:27.187ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
53762025-03-21T02:09:27.187ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
53772025-03-21T02:09:27.187ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-21 02:04:30.886360 UTC dns_group = internal generation = 1
53782025-03-21T02:09:27.187ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 268.408072ms iteration = 6 status = {"generation":1}
53792025-03-21T02:09:27.260ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
53802025-03-21T02:09:27.260ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
53812025-03-21T02:09:27.260ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3d876da-e9f4-4d51-af4c-afd9e21abf2b", "content-length": "26", "date": "Fri, 21 Mar 2025 02:09:26 GMT"} })
53822025-03-21T02:09:27.260ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
53832025-03-21T02:09:27.262ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
53842025-03-21T02:09:27.272ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
53852025-03-21T02:09:27.290ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
53862025-03-21T02:09:27.291ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "53737c40-f1d0-479b-a201-0d0121ceed17", "content-length": "1", "date": "Fri, 21 Mar 2025 02:09:26 GMT"} })
53872025-03-21T02:09:27.292ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
53882025-03-21T02:09:27.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
53892025-03-21T02:09:27.303ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
53902025-03-21T02:09:27.303ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
53912025-03-21T02:09:27.303ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "50d78117-e90e-4172-9736-bb236509003a", "content-length": "372", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} })
53922025-03-21T02:09:27.303ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53932025-03-21T02:09:27.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53942025-03-21T02:09:27.306ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
53952025-03-21T02:09:27.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
53962025-03-21T02:09:27.308ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53972025-03-21T02:09:27.308ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53982025-03-21T02:09:27.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53992025-03-21T02:09:27.310ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 391.532016ms iteration = 11 status = null
54002025-03-21T02:09:27.310ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54012025-03-21T02:09:27.311ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54022025-03-21T02:09:27.311ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
54032025-03-21T02:09:27.311ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54042025-03-21T02:09:27.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54052025-03-21T02:09:27.314ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
54062025-03-21T02:09:27.314ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
54072025-03-21T02:09:27.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
54082025-03-21T02:09:27.315ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
54092025-03-21T02:09:27.315ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54102025-03-21T02:09:27.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54112025-03-21T02:09:27.316ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
54122025-03-21T02:09:27.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
54132025-03-21T02:09:27.318ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
54142025-03-21T02:09:27.319ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54152025-03-21T02:09:27.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54162025-03-21T02:09:27.320ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54172025-03-21T02:09:27.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54182025-03-21T02:09:27.321ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54192025-03-21T02:09:27.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54202025-03-21T02:09:27.322ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54212025-03-21T02:09:27.324ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54222025-03-21T02:09:27.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54232025-03-21T02:09:27.324ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
54242025-03-21T02:09:27.324ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54252025-03-21T02:09:27.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54262025-03-21T02:09:27.325ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 406.781092ms iteration = 11 status = {}
54272025-03-21T02:09:27.330ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54282025-03-21T02:09:27.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54292025-03-21T02:09:27.332ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54302025-03-21T02:09:27.333ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54312025-03-21T02:09:27.350ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54322025-03-21T02:09:27.351ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
54332025-03-21T02:09:27.352ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
54342025-03-21T02:09:27.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
54352025-03-21T02:09:27.353ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54362025-03-21T02:09:27.355ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
54372025-03-21T02:09:27.355ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
54382025-03-21T02:09:27.355ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
54392025-03-21T02:09:27.358ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54402025-03-21T02:09:27.359ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
54412025-03-21T02:09:27.359ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54422025-03-21T02:09:27.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54432025-03-21T02:09:27.362ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
54442025-03-21T02:09:27.365ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 443.080353ms iteration = 3 status = {"blueprint_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","inventory_collection_id":"2e917296-4190-4257-a809-df5c6fd8875a","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
54452025-03-21T02:09:27.365ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54462025-03-21T02:09:27.365ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
54472025-03-21T02:09:27.365ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
54482025-03-21T02:09:27.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54492025-03-21T02:09:27.367ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
54502025-03-21T02:09:27.367ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "3eb7cf61-fc67-49c0-b631-cc3e671f6892", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
54512025-03-21T02:09:27.367ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
54522025-03-21T02:09:27.369ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "c8df4bf1-4f74-4dcf-87c1-748164c8439c", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
54532025-03-21T02:09:27.369ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
54542025-03-21T02:09:27.371ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
54552025-03-21T02:09:27.371ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 452.216972ms iteration = 8 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
54562025-03-21T02:09:27.377ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
54572025-03-21T02:09:27.377ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "8c864af1-c98e-4027-bb37-b301945ede90", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
54582025-03-21T02:09:27.377ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
54592025-03-21T02:09:27.379ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "07b39fda-8c63-4025-8e63-53f71a6d6eb0", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
54602025-03-21T02:09:27.379ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
54612025-03-21T02:09:27.380ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "c8980d2b-fb64-40df-9c22-915fc3990f46", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
54622025-03-21T02:09:27.381ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
54632025-03-21T02:09:27.383ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
54642025-03-21T02:09:27.383ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 464.473975ms iteration = 8 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
54652025-03-21T02:09:27.393ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54662025-03-21T02:09:27.397ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
54672025-03-21T02:09:27.397ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54682025-03-21T02:09:27.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54692025-03-21T02:09:27.399ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54702025-03-21T02:09:27.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54712025-03-21T02:09:27.403ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54722025-03-21T02:09:27.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54732025-03-21T02:09:27.406ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54742025-03-21T02:09:27.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54752025-03-21T02:09:27.409ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54762025-03-21T02:09:27.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54772025-03-21T02:09:27.427ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "cd82cfdf-13fd-43ca-8a98-3704ee782ee3", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} })
54782025-03-21T02:09:27.427ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
54792025-03-21T02:09:27.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
54802025-03-21T02:09:27.427ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc2edf12-8269-492e-b615-6c533c7f9d0e", "content-length": "2", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} })
54812025-03-21T02:09:27.429ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
54822025-03-21T02:09:27.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
54832025-03-21T02:09:27.430ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:615 sled = http://[fd00:1122:3344:101::1]:12345
54842025-03-21T02:09:27.432ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 513.021231ms iteration = 2 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"ad5b28f8-6cad-4221-97f9-e84a46a6381b","time":"2025-03-21T02:09:27.430025209Z"}]}
54852025-03-21T02:09:27.437ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
54862025-03-21T02:09:27.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
54872025-03-21T02:09:27.441ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
54882025-03-21T02:09:27.441ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
54892025-03-21T02:09:27.442ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "81708d7e-0d3f-4684-9e76-56180092f38d", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} })
54902025-03-21T02:09:27.442ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54912025-03-21T02:09:27.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54922025-03-21T02:09:27.444ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 525.356865ms iteration = 11 status = {}
54932025-03-21T02:09:27.466ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54942025-03-21T02:09:27.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54952025-03-21T02:09:27.476ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
54962025-03-21T02:09:27.478ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54972025-03-21T02:09:27.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54982025-03-21T02:09:27.478ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
54992025-03-21T02:09:27.478ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55002025-03-21T02:09:27.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55012025-03-21T02:09:27.479ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55022025-03-21T02:09:27.483ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
55032025-03-21T02:09:27.483ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55042025-03-21T02:09:27.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55052025-03-21T02:09:27.485ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55062025-03-21T02:09:27.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55072025-03-21T02:09:27.486ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55082025-03-21T02:09:27.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55092025-03-21T02:09:27.488ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55102025-03-21T02:09:27.489ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
55112025-03-21T02:09:27.490ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa4807e4-6a72-452a-a11a-a257c496febd", "content-length": "134", "date": "Fri, 21 Mar 2025 02:09:26 GMT"} })
55122025-03-21T02:09:27.490ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
55132025-03-21T02:09:27.490ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
55142025-03-21T02:09:27.490ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
55152025-03-21T02:09:27.490ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
55162025-03-21T02:09:27.490ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
55172025-03-21T02:09:27.490ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
55182025-03-21T02:09:27.490ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
55192025-03-21T02:09:27.490ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
55202025-03-21T02:09:27.490ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
55212025-03-21T02:09:27.490ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
55222025-03-21T02:09:27.490ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
55232025-03-21T02:09:27.490ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55242025-03-21T02:09:27.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55252025-03-21T02:09:27.492ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
55262025-03-21T02:09:27.492ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55272025-03-21T02:09:27.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55282025-03-21T02:09:27.492ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
55292025-03-21T02:09:27.493ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf662e1a-6efc-4fef-9db8-fc39cc88fa6d", "content-length": "2", "date": "Fri, 21 Mar 2025 02:09:26 GMT"} })
55302025-03-21T02:09:27.493ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
55312025-03-21T02:09:27.493ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "cba0bc4b-946b-407c-93d0-95b5c89773d2", "content-length": "52", "date": "Fri, 21 Mar 2025 02:09:26 GMT"} })
55322025-03-21T02:09:27.493ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
55332025-03-21T02:09:27.493ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
55342025-03-21T02:09:27.496ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55352025-03-21T02:09:27.496ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55362025-03-21T02:09:27.498ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55372025-03-21T02:09:27.498ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55382025-03-21T02:09:27.500ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55392025-03-21T02:09:27.500ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55402025-03-21T02:09:27.501ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55412025-03-21T02:09:27.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55422025-03-21T02:09:27.504ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55432025-03-21T02:09:27.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55442025-03-21T02:09:27.505ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55452025-03-21T02:09:27.505ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55462025-03-21T02:09:27.507ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55472025-03-21T02:09:27.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55482025-03-21T02:09:27.508ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55492025-03-21T02:09:27.512ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
55502025-03-21T02:09:27.512ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55512025-03-21T02:09:27.512ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55522025-03-21T02:09:27.513ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55532025-03-21T02:09:27.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55542025-03-21T02:09:27.515ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55552025-03-21T02:09:27.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55562025-03-21T02:09:27.516ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55572025-03-21T02:09:27.520ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
55582025-03-21T02:09:27.520ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55592025-03-21T02:09:27.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55602025-03-21T02:09:27.534ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 5a5beddc-544e-4e62-bf20-c61d9dde2785, kind: Service { id: bfbad16c-d6a4-4742-9db2-139d724c0771 }, name: Name("external-dns-bfbad16c-d6a4-4742-9db2-139d724c0771"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 221])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bfb6f529-ad0a-4d8f-8e27-90ee0b561301, kind: Service { id: a1777e88-1a9c-49b1-bca1-ec6c5e1991fc }, name: Name("external-dns-a1777e88-1a9c-49b1-bca1-ec6c5e1991fc"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 183, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 34365a8b-47e5-4623-9ff4-fa54efca49f0, kind: Service { id: 6021c9d9-927c-44a6-8331-6ab48efd9c00 }, name: Name("nexus-6021c9d9-927c-44a6-8331-6ab48efd9c00"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 210])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8d8a5138-cad9-4d17-aa4a-561116e6b6ee, kind: Service { id: e05eb799-8cf0-4702-bd3b-9994ae86118a }, name: Name("nexus-e05eb799-8cf0-4702-bd3b-9994ae86118a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 112])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c4619e25-4546-4262-8518-a5e7bdb39205, kind: Service { id: 4b42f969-b761-4b99-9019-b41841cb0f60 }, name: Name("nexus-4b42f969-b761-4b99-9019-b41841cb0f60"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 128])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
55612025-03-21T02:09:27.534ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55622025-03-21T02:09:27.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55632025-03-21T02:09:27.536ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55642025-03-21T02:09:27.537ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
55652025-03-21T02:09:27.537ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55662025-03-21T02:09:27.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55672025-03-21T02:09:27.546ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
55682025-03-21T02:09:27.546ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 5a5beddc-544e-4e62-bf20-c61d9dde2785, kind: Service { id: bfbad16c-d6a4-4742-9db2-139d724c0771 }, name: Name("external-dns-bfbad16c-d6a4-4742-9db2-139d724c0771"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 221])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bfb6f529-ad0a-4d8f-8e27-90ee0b561301, kind: Service { id: a1777e88-1a9c-49b1-bca1-ec6c5e1991fc }, name: Name("external-dns-a1777e88-1a9c-49b1-bca1-ec6c5e1991fc"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 183, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 34365a8b-47e5-4623-9ff4-fa54efca49f0, kind: Service { id: 6021c9d9-927c-44a6-8331-6ab48efd9c00 }, name: Name("nexus-6021c9d9-927c-44a6-8331-6ab48efd9c00"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 210])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8d8a5138-cad9-4d17-aa4a-561116e6b6ee, kind: Service { id: e05eb799-8cf0-4702-bd3b-9994ae86118a }, name: Name("nexus-e05eb799-8cf0-4702-bd3b-9994ae86118a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 112])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c4619e25-4546-4262-8518-a5e7bdb39205, kind: Service { id: 4b42f969-b761-4b99-9019-b41841cb0f60 }, name: Name("nexus-4b42f969-b761-4b99-9019-b41841cb0f60"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 128])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
55692025-03-21T02:09:27.546ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
55702025-03-21T02:09:27.577ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d179899-87b1-4bf2-a2bb-6455806d637d", "content-length": "691", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} })
55712025-03-21T02:09:27.577ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
55722025-03-21T02:09:27.577ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
55732025-03-21T02:09:27.577ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
55742025-03-21T02:09:27.579ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 660.465768ms iteration = 11 status = {}
55752025-03-21T02:09:27.627ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: ad5b28f8-6cad-4221-97f9-e84a46a6381b, time_created: 2025-03-21T02:04:28.969535Z, time_modified: 2025-03-21T02:04:28.969535Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, is_scrimlet: true, serial_number: "buskin", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(56094621696)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
55762025-03-21T02:09:27.629ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
55772025-03-21T02:09:27.631ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55782025-03-21T02:09:27.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55792025-03-21T02:09:27.634ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55802025-03-21T02:09:27.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55812025-03-21T02:09:27.636ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55822025-03-21T02:09:27.640ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
55832025-03-21T02:09:27.671ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
55842025-03-21T02:09:27.672ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "f8a6e997-f292-453c-8e61-acca5631141c", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} })
55852025-03-21T02:09:27.674ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
55862025-03-21T02:09:27.674ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 755.990751ms
55872025-03-21T02:09:27.674ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 756.008911ms iteration = 2 status = {}
55882025-03-21T02:09:27.677ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "b16331c7-a3ab-4ba9-8cab-7b8991a0e984", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} })
55892025-03-21T02:09:27.677ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
55902025-03-21T02:09:27.677ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
55912025-03-21T02:09:27.677ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b50dc78-4a01-476a-8aec-1ee5e1ba7191", "content-length": "682", "date": "Fri, 21 Mar 2025 02:09:27 GMT"} })
55922025-03-21T02:09:27.678ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55932025-03-21T02:09:27.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55942025-03-21T02:09:27.680ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55952025-03-21T02:09:27.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55962025-03-21T02:09:27.683ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55972025-03-21T02:09:27.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55982025-03-21T02:09:27.685ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55992025-03-21T02:09:27.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56002025-03-21T02:09:27.693ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 774.129227ms iteration = 11 status = {}
56012025-03-21T02:09:29.719ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 325 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = a5f56fef-457b-4712-9032-b9bd0969c002 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
56022025-03-21T02:09:36.918ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 32 reason = Timeout
56032025-03-21T02:09:36.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56042025-03-21T02:09:36.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
56052025-03-21T02:09:36.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56062025-03-21T02:09:36.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56072025-03-21T02:09:36.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56082025-03-21T02:09:36.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56092025-03-21T02:09:36.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56102025-03-21T02:09:36.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56112025-03-21T02:09:36.923ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
56122025-03-21T02:09:36.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56132025-03-21T02:09:36.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56142025-03-21T02:09:36.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56152025-03-21T02:09:36.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56162025-03-21T02:09:36.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56172025-03-21T02:09:36.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
56182025-03-21T02:09:36.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56192025-03-21T02:09:36.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56202025-03-21T02:09:36.939ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
56212025-03-21T02:09:36.939ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.506926ms iteration = 32 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
56222025-03-21T02:09:36.952ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.527954ms iteration = 32 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
56232025-03-21T02:09:39.719ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 303 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 0a1dfd1b-d139-4845-9ba8-47df540980b2 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
56242025-03-21T02:09:46.919ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 33 reason = Timeout
56252025-03-21T02:09:46.919ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56262025-03-21T02:09:46.919ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
56272025-03-21T02:09:46.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56282025-03-21T02:09:46.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56292025-03-21T02:09:46.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56302025-03-21T02:09:46.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56312025-03-21T02:09:46.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56322025-03-21T02:09:46.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56332025-03-21T02:09:46.923ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
56342025-03-21T02:09:46.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56352025-03-21T02:09:46.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56362025-03-21T02:09:46.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56372025-03-21T02:09:46.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56382025-03-21T02:09:46.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56392025-03-21T02:09:46.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
56402025-03-21T02:09:46.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56412025-03-21T02:09:46.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56422025-03-21T02:09:46.937ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
56432025-03-21T02:09:46.937ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.471791ms iteration = 33 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
56442025-03-21T02:09:46.950ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.01862ms iteration = 33 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
56452025-03-21T02:09:49.719ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 449 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = ff106b5f-7f27-4ec7-91bb-5d33d6422a1a response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
56462025-03-21T02:09:54.710ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:51080
56472025-03-21T02:09:54.710ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51080 req_id = 41401a42-9743-4c93-849c-aeb14d57d0c3 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
56482025-03-21T02:09:54.710ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51080 req_id = 41401a42-9743-4c93-849c-aeb14d57d0c3 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
56492025-03-21T02:09:54.713ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3360 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51080 req_id = 41401a42-9743-4c93-849c-aeb14d57d0c3 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
56502025-03-21T02:09:54.714ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51080 req_id = 19c5c1c7-fbe4-4719-966a-a2578f3900b3 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
56512025-03-21T02:09:54.714ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51080 req_id = 19c5c1c7-fbe4-4719-966a-a2578f3900b3 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
56522025-03-21T02:09:54.716ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2296 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51080 req_id = 19c5c1c7-fbe4-4719-966a-a2578f3900b3 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
56532025-03-21T02:09:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 34 reason = Timeout
56542025-03-21T02:09:56.918ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
56552025-03-21T02:09:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56562025-03-21T02:09:56.918ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
56572025-03-21T02:09:56.918ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56582025-03-21T02:09:56.918ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
56592025-03-21T02:09:56.918ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
56602025-03-21T02:09:56.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
56612025-03-21T02:09:56.918ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
56622025-03-21T02:09:56.918ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
56632025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
56642025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
56652025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56662025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 12 reason = Timeout
56672025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 12 reason = Timeout
56682025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
56692025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
56702025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 12 reason = Timeout
56712025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56722025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 12 reason = Timeout
56732025-03-21T02:09:56.919ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
56742025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56752025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 12 reason = Timeout
56762025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
56772025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
56782025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 12 reason = Timeout
56792025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
56802025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 12 reason = Timeout
56812025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
56822025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 12 reason = Timeout
56832025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
56842025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 13 reason = Timeout
56852025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56862025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 12 reason = Timeout
56872025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
56882025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 12 reason = Timeout
56892025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56902025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 12 reason = Timeout
56912025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
56922025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56932025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
56942025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
56952025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56962025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
56972025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
56982025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
56992025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
57002025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
57012025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57022025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
57032025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
57042025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57052025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
57062025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
57072025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
57082025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
57092025-03-21T02:09:56.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57102025-03-21T02:09:56.920ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57112025-03-21T02:09:56.922ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1305, tv_nsec: 545609623 } })
57122025-03-21T02:09:56.922ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1305, tv_nsec: 545726505 } })
57132025-03-21T02:09:56.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
57142025-03-21T02:09:56.923ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
57152025-03-21T02:09:56.923ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
57162025-03-21T02:09:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
57172025-03-21T02:09:56.923ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.438358ms iteration = 12 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
57182025-03-21T02:09:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
57192025-03-21T02:09:56.923ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57202025-03-21T02:09:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57212025-03-21T02:09:56.925ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57222025-03-21T02:09:56.926ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57232025-03-21T02:09:56.926ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57242025-03-21T02:09:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
57252025-03-21T02:09:56.928ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
57262025-03-21T02:09:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
57272025-03-21T02:09:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
57282025-03-21T02:09:56.928ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57292025-03-21T02:09:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
57302025-03-21T02:09:56.928ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
57312025-03-21T02:09:56.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57322025-03-21T02:09:56.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
57332025-03-21T02:09:56.930ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57342025-03-21T02:09:56.930ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57352025-03-21T02:09:56.931ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
57362025-03-21T02:09:56.931ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57372025-03-21T02:09:56.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57382025-03-21T02:09:56.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57392025-03-21T02:09:56.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
57402025-03-21T02:09:56.933ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
57412025-03-21T02:09:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
57422025-03-21T02:09:56.935ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
57432025-03-21T02:09:56.935ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57442025-03-21T02:09:56.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57452025-03-21T02:09:56.942ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57462025-03-21T02:09:56.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57472025-03-21T02:09:56.945ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1305, tv_nsec: 568721396 } })
57482025-03-21T02:09:56.946ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
57492025-03-21T02:09:56.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57502025-03-21T02:09:56.946ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
57512025-03-21T02:09:56.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57522025-03-21T02:09:56.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
57532025-03-21T02:09:56.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57542025-03-21T02:09:56.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
57552025-03-21T02:09:56.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57562025-03-21T02:09:56.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57572025-03-21T02:09:56.962ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
57582025-03-21T02:09:56.962ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 43.852644ms iteration = 34 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
57592025-03-21T02:09:56.973ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
57602025-03-21T02:09:56.973ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
57612025-03-21T02:09:56.973ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
57622025-03-21T02:09:56.973ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
57632025-03-21T02:09:56.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
57642025-03-21T02:09:56.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
57652025-03-21T02:09:56.974ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
57662025-03-21T02:09:56.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
57672025-03-21T02:09:56.975ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 56.048776ms iteration = 12 status = {"errors":[],"garbage_collect_requested":[]}
57682025-03-21T02:09:56.976ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
57692025-03-21T02:09:56.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
57702025-03-21T02:09:56.977ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 59.013477ms iteration = 12 status = {"errors":[],"finish_invoked_ok":[]}
57712025-03-21T02:09:56.980ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 62.01357ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
57722025-03-21T02:09:56.985ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.899585ms iteration = 34 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
57732025-03-21T02:09:56.986ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
57742025-03-21T02:09:56.986ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57752025-03-21T02:09:56.986ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "09b0213b-7560-43b4-9c2b-0e5c29992d32", "content-length": "26", "date": "Fri, 21 Mar 2025 02:09:56 GMT"} })
57762025-03-21T02:09:56.986ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57772025-03-21T02:09:56.988ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 68.978551ms iteration = 12 status = {"errors":[],"requests_created_ok":[]}
57782025-03-21T02:09:56.988ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
57792025-03-21T02:09:56.990ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
57802025-03-21T02:09:56.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
57812025-03-21T02:09:56.992ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57822025-03-21T02:09:56.992ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
57832025-03-21T02:09:56.992ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57842025-03-21T02:09:56.992ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "eaac1673-8def-482c-acab-a68cefd01dc8", "content-length": "26", "date": "Fri, 21 Mar 2025 02:09:56 GMT"} })
57852025-03-21T02:09:56.992ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57862025-03-21T02:09:56.994ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
57872025-03-21T02:09:56.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
57882025-03-21T02:09:56.995ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
57892025-03-21T02:09:56.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
57902025-03-21T02:09:56.995ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57912025-03-21T02:09:56.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
57922025-03-21T02:09:56.998ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
57932025-03-21T02:09:56.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
57942025-03-21T02:09:56.998ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57952025-03-21T02:09:56.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57962025-03-21T02:09:56.999ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57972025-03-21T02:09:57.001ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57982025-03-21T02:09:57.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57992025-03-21T02:09:57.002ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
58002025-03-21T02:09:57.002ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58012025-03-21T02:09:57.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58022025-03-21T02:09:57.004ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58032025-03-21T02:09:57.005ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
58042025-03-21T02:09:57.005ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 86.508086ms iteration = 12 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
58052025-03-21T02:09:57.008ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
58062025-03-21T02:09:57.008ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
58072025-03-21T02:09:57.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
58082025-03-21T02:09:57.010ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 91.580334ms iteration = 12 status = 0
58092025-03-21T02:09:57.014ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 95.799428ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
58102025-03-21T02:09:57.022ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
58112025-03-21T02:09:57.023ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
58122025-03-21T02:09:57.023ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
58132025-03-21T02:09:57.023ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
58142025-03-21T02:09:57.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
58152025-03-21T02:09:57.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
58162025-03-21T02:09:57.032ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58172025-03-21T02:09:57.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58182025-03-21T02:09:57.033ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
58192025-03-21T02:09:57.033ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
58202025-03-21T02:09:57.033ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "769a3cdd-214a-4944-b520-8697ff89398e", "content-length": "26", "date": "Fri, 21 Mar 2025 02:09:56 GMT"} })
58212025-03-21T02:09:57.033ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
58222025-03-21T02:09:57.035ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
58232025-03-21T02:09:57.035ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
58242025-03-21T02:09:57.035ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 116.729153ms iteration = 12 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
58252025-03-21T02:09:57.054ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
58262025-03-21T02:09:57.054ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
58272025-03-21T02:09:57.054ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 135.723543ms iteration = 12 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
58282025-03-21T02:09:57.055ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
58292025-03-21T02:09:57.056ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b97539b-ec59-4187-9a48-e5615575cb40", "content-length": "1", "date": "Fri, 21 Mar 2025 02:09:56 GMT"} })
58302025-03-21T02:09:57.058ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
58312025-03-21T02:09:57.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
58322025-03-21T02:09:57.061ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 142.236416ms iteration = 12 status = {}
58332025-03-21T02:09:57.061ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 142.554802ms iteration = 12 status = null
58342025-03-21T02:09:57.066ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 147.141682ms iteration = 12 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
58352025-03-21T02:09:57.106ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 186.960125ms iteration = 13 status = {"physical_disks_added":0}
58362025-03-21T02:09:57.109ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
58372025-03-21T02:09:57.109ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
58382025-03-21T02:09:57.120ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d91132c-b4f1-4cf2-8a30-37da4af7c670", "content-length": "372", "date": "Fri, 21 Mar 2025 02:09:56 GMT"} })
58392025-03-21T02:09:57.120ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58402025-03-21T02:09:57.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58412025-03-21T02:09:57.134ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58422025-03-21T02:09:57.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58432025-03-21T02:09:57.135ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
58442025-03-21T02:09:57.135ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
58452025-03-21T02:09:57.135ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "15d93d78-e2dc-43d0-a753-2536820f15ad", "date": "Fri, 21 Mar 2025 02:09:56 GMT"} })
58462025-03-21T02:09:57.135ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58472025-03-21T02:09:57.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58482025-03-21T02:09:57.136ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58492025-03-21T02:09:57.137ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
58502025-03-21T02:09:57.137ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58512025-03-21T02:09:57.137ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58522025-03-21T02:09:57.138ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58532025-03-21T02:09:57.138ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58542025-03-21T02:09:57.139ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58552025-03-21T02:09:57.141ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
58562025-03-21T02:09:57.141ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58572025-03-21T02:09:57.141ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58582025-03-21T02:09:57.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58592025-03-21T02:09:57.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58602025-03-21T02:09:57.144ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
58612025-03-21T02:09:57.145ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
58622025-03-21T02:09:57.145ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "7accec39-1cf3-4600-abd8-025cc9118f9c", "date": "Fri, 21 Mar 2025 02:09:56 GMT"} })
58632025-03-21T02:09:57.146ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 227.609202ms iteration = 12 status = {}
58642025-03-21T02:09:57.148ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58652025-03-21T02:09:57.148ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58662025-03-21T02:09:57.150ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
58672025-03-21T02:09:57.150ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58682025-03-21T02:09:57.150ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58692025-03-21T02:09:57.160ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
58702025-03-21T02:09:57.160ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "6391ecbe-8c49-414d-9355-80469c09dea4", "content-length": "134", "date": "Fri, 21 Mar 2025 02:09:56 GMT"} })
58712025-03-21T02:09:57.160ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
58722025-03-21T02:09:57.160ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
58732025-03-21T02:09:57.161ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
58742025-03-21T02:09:57.161ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
58752025-03-21T02:09:57.161ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58762025-03-21T02:09:57.161ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58772025-03-21T02:09:57.161ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58782025-03-21T02:09:57.161ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
58792025-03-21T02:09:57.161ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58802025-03-21T02:09:57.161ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
58812025-03-21T02:09:57.161ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
58822025-03-21T02:09:57.161ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58832025-03-21T02:09:57.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58842025-03-21T02:09:57.163ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
58852025-03-21T02:09:57.163ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e7c8272-c3b5-4a7d-9d05-a61823d5d543", "content-length": "2", "date": "Fri, 21 Mar 2025 02:09:56 GMT"} })
58862025-03-21T02:09:57.163ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
58872025-03-21T02:09:57.163ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "f76009f1-2661-417c-8f9e-7b5382112f2e", "content-length": "52", "date": "Fri, 21 Mar 2025 02:09:56 GMT"} })
58882025-03-21T02:09:57.163ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
58892025-03-21T02:09:57.163ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
58902025-03-21T02:09:57.177ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
58912025-03-21T02:09:57.177ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "8623863a-9a3a-4bb8-b71e-1ff264d50ddb", "content-length": "691", "date": "Fri, 21 Mar 2025 02:09:56 GMT"} })
58922025-03-21T02:09:57.177ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
58932025-03-21T02:09:57.177ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
58942025-03-21T02:09:57.177ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
58952025-03-21T02:09:57.178ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 259.929306ms iteration = 12 status = {}
58962025-03-21T02:09:57.257ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "9c397695-fd08-44cb-8ee4-c1b24eca38d2", "date": "Fri, 21 Mar 2025 02:09:57 GMT"} })
58972025-03-21T02:09:57.257ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
58982025-03-21T02:09:57.257ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
58992025-03-21T02:09:57.258ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f3e040c-326a-48d5-8de1-3f5912023985", "content-length": "682", "date": "Fri, 21 Mar 2025 02:09:56 GMT"} })
59002025-03-21T02:09:57.258ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59012025-03-21T02:09:57.258ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59022025-03-21T02:09:57.262ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59032025-03-21T02:09:57.262ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59042025-03-21T02:09:57.265ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59052025-03-21T02:09:57.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59062025-03-21T02:09:57.267ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59072025-03-21T02:09:57.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59082025-03-21T02:09:57.273ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 354.439501ms iteration = 12 status = {}
59092025-03-21T02:09:59.719ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = f48082f7-c45c-44db-8366-7c9ee4ba51c6 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
59102025-03-21T02:10:06.919ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
59112025-03-21T02:10:06.919ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 35 reason = Timeout
59122025-03-21T02:10:06.919ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59132025-03-21T02:10:06.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59142025-03-21T02:10:06.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59152025-03-21T02:10:06.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59162025-03-21T02:10:06.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59172025-03-21T02:10:06.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59182025-03-21T02:10:06.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59192025-03-21T02:10:06.923ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.927438ms iteration = 35 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
59202025-03-21T02:10:06.923ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
59212025-03-21T02:10:06.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59222025-03-21T02:10:06.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59232025-03-21T02:10:06.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59242025-03-21T02:10:06.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59252025-03-21T02:10:06.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59262025-03-21T02:10:06.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
59272025-03-21T02:10:06.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59282025-03-21T02:10:06.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59292025-03-21T02:10:06.935ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
59302025-03-21T02:10:06.935ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.292163ms iteration = 35 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
59312025-03-21T02:10:09.719ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 279 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 1c2f6bec-d39c-42fd-a0ba-ad16f951c390 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
59322025-03-21T02:10:16.919ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 36 reason = Timeout
59332025-03-21T02:10:16.919ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59342025-03-21T02:10:16.919ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
59352025-03-21T02:10:16.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59362025-03-21T02:10:16.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59372025-03-21T02:10:16.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59382025-03-21T02:10:16.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59392025-03-21T02:10:16.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59402025-03-21T02:10:16.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59412025-03-21T02:10:16.924ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
59422025-03-21T02:10:16.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59432025-03-21T02:10:16.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59442025-03-21T02:10:16.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59452025-03-21T02:10:16.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59462025-03-21T02:10:16.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59472025-03-21T02:10:16.931ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
59482025-03-21T02:10:16.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59492025-03-21T02:10:16.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59502025-03-21T02:10:16.939ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
59512025-03-21T02:10:16.939ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.049289ms iteration = 36 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
59522025-03-21T02:10:16.952ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.697569ms iteration = 36 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
59532025-03-21T02:10:19.719ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 292 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 4fe4e89d-596a-4593-81b2-8e2e8eb30d93 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
59542025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 37 reason = Timeout
59552025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59562025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
59572025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 4 reason = Timeout
59582025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
59592025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 13 reason = Timeout
59602025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
59612025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
59622025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 13 reason = Timeout
59632025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
59642025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 13 reason = Timeout
59652025-03-21T02:10:26.919ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
59662025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 7 reason = Timeout
59672025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
59682025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 13 reason = Timeout
59692025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59702025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
59712025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 13 reason = Timeout
59722025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
59732025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 13 reason = Timeout
59742025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
59752025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
59762025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 7 reason = Timeout
59772025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
59782025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
59792025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
59802025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59812025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59822025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
59832025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
59842025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
59852025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
59862025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
59872025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
59882025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 7 reason = Timeout
59892025-03-21T02:10:26.919ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 02:00:26.919367816 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
59902025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
59912025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
59922025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
59932025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
59942025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 13 reason = Timeout
59952025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
59962025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
59972025-03-21T02:10:26.919ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
59982025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 7 reason = Timeout
59992025-03-21T02:10:26.919ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
60002025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 16.08\u{b5}s iteration = 8 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
60012025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
60022025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
60032025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 13 reason = Timeout
60042025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
60052025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 13 reason = Timeout
60062025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
60072025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
60082025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 7 reason = Timeout
60092025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
60102025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 13 reason = Timeout
60112025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
60122025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
60132025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
60142025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
60152025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 13 reason = Timeout
60162025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
60172025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
60182025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
60192025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 14 reason = Timeout
60202025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60212025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
60222025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60232025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
60242025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
60252025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60262025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
60272025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
60282025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60292025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
60302025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60312025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
60322025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
60332025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
60342025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
60352025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60362025-03-21T02:10:26.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60372025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60382025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
60392025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
60402025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
60412025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
60422025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
60432025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60442025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60452025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
60462025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
60472025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60482025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
60492025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
60502025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
60512025-03-21T02:10:26.920ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1335, tv_nsec: 542706867 } })
60522025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
60532025-03-21T02:10:26.920ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1335, tv_nsec: 542773718 } })
60542025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60552025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
60562025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60572025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
60582025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
60592025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.280582ms iteration = 7 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
60602025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60612025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
60622025-03-21T02:10:26.920ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.390244ms iteration = 7 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
60632025-03-21T02:10:26.922ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60642025-03-21T02:10:26.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
60652025-03-21T02:10:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60662025-03-21T02:10:26.923ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.246454ms iteration = 13 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
60672025-03-21T02:10:26.924ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1335, tv_nsec: 546963721 } })
60682025-03-21T02:10:26.924ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60692025-03-21T02:10:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
60702025-03-21T02:10:26.924ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
60712025-03-21T02:10:26.924ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
60722025-03-21T02:10:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
60732025-03-21T02:10:26.926ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60742025-03-21T02:10:26.926ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
60752025-03-21T02:10:26.926ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
60762025-03-21T02:10:26.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60772025-03-21T02:10:26.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
60782025-03-21T02:10:26.927ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
60792025-03-21T02:10:26.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60802025-03-21T02:10:26.928ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
60812025-03-21T02:10:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
60822025-03-21T02:10:26.929ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 02:00:26.919367816 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
60832025-03-21T02:10:26.929ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 9.951023ms iteration = 7 status = {"expiration":"2025-03-21T02:00:26.919367816Z","pruned":[]}
60842025-03-21T02:10:26.931ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 11.996099ms iteration = 7 status = {"errors":[],"found_port_ok":[]}
60852025-03-21T02:10:26.931ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60862025-03-21T02:10:26.931ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60872025-03-21T02:10:26.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60882025-03-21T02:10:26.933ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
60892025-03-21T02:10:26.933ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60902025-03-21T02:10:26.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60912025-03-21T02:10:26.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
60922025-03-21T02:10:26.934ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60932025-03-21T02:10:26.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60942025-03-21T02:10:26.934ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
60952025-03-21T02:10:26.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
60962025-03-21T02:10:26.936ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
60972025-03-21T02:10:26.936ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60982025-03-21T02:10:26.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60992025-03-21T02:10:26.941ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 22.279897ms iteration = 13 status = {"errors":[],"finish_invoked_ok":[]}
61002025-03-21T02:10:26.942ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1335, tv_nsec: 565161778 } })
61012025-03-21T02:10:26.942ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
61022025-03-21T02:10:26.948ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61032025-03-21T02:10:26.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
61042025-03-21T02:10:26.951ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
61052025-03-21T02:10:26.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
61062025-03-21T02:10:26.955ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61072025-03-21T02:10:26.957ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
61082025-03-21T02:10:26.958ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
61092025-03-21T02:10:26.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
61102025-03-21T02:10:26.962ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61112025-03-21T02:10:26.963ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
61122025-03-21T02:10:26.963ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 43.945375ms iteration = 13 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
61132025-03-21T02:10:26.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61142025-03-21T02:10:26.974ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61152025-03-21T02:10:26.977ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61162025-03-21T02:10:26.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
61172025-03-21T02:10:26.978ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
61182025-03-21T02:10:26.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
61192025-03-21T02:10:26.979ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
61202025-03-21T02:10:26.979ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
61212025-03-21T02:10:26.979ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61222025-03-21T02:10:26.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
61232025-03-21T02:10:26.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61242025-03-21T02:10:26.982ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
61252025-03-21T02:10:26.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61262025-03-21T02:10:26.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61272025-03-21T02:10:26.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61282025-03-21T02:10:26.984ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
61292025-03-21T02:10:26.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
61302025-03-21T02:10:26.986ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
61312025-03-21T02:10:26.986ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
61322025-03-21T02:10:26.986ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
61332025-03-21T02:10:26.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
61342025-03-21T02:10:26.989ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61352025-03-21T02:10:26.990ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61362025-03-21T02:10:26.991ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
61372025-03-21T02:10:26.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
61382025-03-21T02:10:26.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
61392025-03-21T02:10:26.992ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 73.714033ms iteration = 13 status = {"errors":[],"garbage_collect_requested":[]}
61402025-03-21T02:10:26.993ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
61412025-03-21T02:10:26.993ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
61422025-03-21T02:10:26.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
61432025-03-21T02:10:26.995ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 75.557655ms iteration = 13 status = {"errors":[],"requests_created_ok":[]}
61442025-03-21T02:10:26.995ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 76.167386ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
61452025-03-21T02:10:26.995ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 76.800997ms iteration = 37 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
61462025-03-21T02:10:26.996ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61472025-03-21T02:10:26.996ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61482025-03-21T02:10:26.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61492025-03-21T02:10:26.997ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61502025-03-21T02:10:26.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
61512025-03-21T02:10:26.998ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
61522025-03-21T02:10:26.998ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61532025-03-21T02:10:26.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
61542025-03-21T02:10:26.999ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61552025-03-21T02:10:27.000ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61562025-03-21T02:10:27.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
61572025-03-21T02:10:27.001ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61582025-03-21T02:10:27.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61592025-03-21T02:10:27.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
61602025-03-21T02:10:27.001ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
61612025-03-21T02:10:27.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
61622025-03-21T02:10:27.001ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61632025-03-21T02:10:27.001ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
61642025-03-21T02:10:27.001ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
61652025-03-21T02:10:27.001ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 82.460746ms iteration = 7 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
61662025-03-21T02:10:27.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
61672025-03-21T02:10:27.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61682025-03-21T02:10:27.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
61692025-03-21T02:10:27.003ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
61702025-03-21T02:10:27.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
61712025-03-21T02:10:27.004ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61722025-03-21T02:10:27.005ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 86.354403ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
61732025-03-21T02:10:27.006ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
61742025-03-21T02:10:27.006ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
61752025-03-21T02:10:27.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
61762025-03-21T02:10:27.009ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61772025-03-21T02:10:27.009ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
61782025-03-21T02:10:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
61792025-03-21T02:10:27.010ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1335, tv_nsec: 633352815 } })
61802025-03-21T02:10:27.011ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
61812025-03-21T02:10:27.011ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
61822025-03-21T02:10:27.011ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61832025-03-21T02:10:27.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61842025-03-21T02:10:27.011ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
61852025-03-21T02:10:27.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
61862025-03-21T02:10:27.012ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
61872025-03-21T02:10:27.012ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
61882025-03-21T02:10:27.012ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
61892025-03-21T02:10:27.012ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
61902025-03-21T02:10:27.012ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
61912025-03-21T02:10:27.012ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
61922025-03-21T02:10:27.012ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 93.04279ms iteration = 7 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
61932025-03-21T02:10:27.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
61942025-03-21T02:10:27.014ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 95.005024ms iteration = 4 status = {"nsuccess":5}
61952025-03-21T02:10:27.014ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61962025-03-21T02:10:27.014ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61972025-03-21T02:10:27.015ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61982025-03-21T02:10:27.017ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
61992025-03-21T02:10:27.017ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
62002025-03-21T02:10:27.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
62012025-03-21T02:10:27.019ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
62022025-03-21T02:10:27.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
62032025-03-21T02:10:27.020ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
62042025-03-21T02:10:27.020ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
62052025-03-21T02:10:27.020ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
62062025-03-21T02:10:27.020ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62072025-03-21T02:10:27.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62082025-03-21T02:10:27.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
62092025-03-21T02:10:27.020ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
62102025-03-21T02:10:27.020ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 101.490827ms iteration = 37 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
62112025-03-21T02:10:27.022ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62122025-03-21T02:10:27.023ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
62132025-03-21T02:10:27.023ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
62142025-03-21T02:10:27.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
62152025-03-21T02:10:27.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
62162025-03-21T02:10:27.025ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62172025-03-21T02:10:27.025ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
62182025-03-21T02:10:27.025ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
62192025-03-21T02:10:27.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
62202025-03-21T02:10:27.026ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
62212025-03-21T02:10:27.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
62222025-03-21T02:10:27.027ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
62232025-03-21T02:10:27.027ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
62242025-03-21T02:10:27.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
62252025-03-21T02:10:27.028ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62262025-03-21T02:10:27.031ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
62272025-03-21T02:10:27.031ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
62282025-03-21T02:10:27.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
62292025-03-21T02:10:27.032ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 113.632608ms iteration = 13 status = 0
62302025-03-21T02:10:27.040ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
62312025-03-21T02:10:27.040ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
62322025-03-21T02:10:27.040ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
62332025-03-21T02:10:27.040ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
62342025-03-21T02:10:27.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
62352025-03-21T02:10:27.042ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
62362025-03-21T02:10:27.042ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
62372025-03-21T02:10:27.042ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
62382025-03-21T02:10:27.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
62392025-03-21T02:10:27.042ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62402025-03-21T02:10:27.044ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62412025-03-21T02:10:27.046ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
62422025-03-21T02:10:27.046ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
62432025-03-21T02:10:27.046ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
62442025-03-21T02:10:27.046ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
62452025-03-21T02:10:27.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
62462025-03-21T02:10:27.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
62472025-03-21T02:10:27.057ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
62482025-03-21T02:10:27.058ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
62492025-03-21T02:10:27.058ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
62502025-03-21T02:10:27.058ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62512025-03-21T02:10:27.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
62522025-03-21T02:10:27.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62532025-03-21T02:10:27.060ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62542025-03-21T02:10:27.064ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
62552025-03-21T02:10:27.064ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62562025-03-21T02:10:27.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62572025-03-21T02:10:27.065ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 146.113854ms iteration = 13 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
62582025-03-21T02:10:27.067ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
62592025-03-21T02:10:27.067ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
62602025-03-21T02:10:27.067ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-21 02:04:30.886360 UTC dns_group = internal generation = 1
62612025-03-21T02:10:27.067ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 148.418733ms iteration = 7 status = {"generation":1}
62622025-03-21T02:10:27.069ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
62632025-03-21T02:10:27.069ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 150.46662ms iteration = 13 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
62642025-03-21T02:10:27.094ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
62652025-03-21T02:10:27.094ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
62662025-03-21T02:10:27.094ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 175.064978ms iteration = 13 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
62672025-03-21T02:10:27.094ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) }
62682025-03-21T02:10:27.094ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 175.261842ms iteration = 7 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
62692025-03-21T02:10:27.096ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 177.148514ms iteration = 14 status = {"physical_disks_added":0}
62702025-03-21T02:10:27.104ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
62712025-03-21T02:10:27.104ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
62722025-03-21T02:10:27.104ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
62732025-03-21T02:10:27.104ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
62742025-03-21T02:10:27.106ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62752025-03-21T02:10:27.108ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
62762025-03-21T02:10:27.108ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
62772025-03-21T02:10:27.109ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
62782025-03-21T02:10:27.110ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
62792025-03-21T02:10:27.110ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
62802025-03-21T02:10:27.110ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-21 02:04:31.662541 UTC dns_group = external generation = 2
62812025-03-21T02:10:27.110ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 191.508264ms iteration = 7 status = {"generation":2}
62822025-03-21T02:10:27.120ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
62832025-03-21T02:10:27.128ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
62842025-03-21T02:10:27.128ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 209.532188ms iteration = 7 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
62852025-03-21T02:10:27.192ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
62862025-03-21T02:10:27.192ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
62872025-03-21T02:10:27.287ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f6d9866-ba72-490d-9255-686e8c8a00e6", "content-length": "372", "date": "Fri, 21 Mar 2025 02:10:27 GMT"} })
62882025-03-21T02:10:27.287ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
62892025-03-21T02:10:27.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
62902025-03-21T02:10:27.303ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
62912025-03-21T02:10:27.315ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
62922025-03-21T02:10:27.316ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a36ad95-645a-46fd-9ab7-a594efdb16da", "content-length": "691", "date": "Fri, 21 Mar 2025 02:10:27 GMT"} })
62932025-03-21T02:10:27.316ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
62942025-03-21T02:10:27.316ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
62952025-03-21T02:10:27.316ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
62962025-03-21T02:10:27.318ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 398.601979ms iteration = 13 status = {}
62972025-03-21T02:10:27.326ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
62982025-03-21T02:10:27.326ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
62992025-03-21T02:10:27.326ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "aafc5030-93c7-4c57-82ce-5c2ec822c716", "content-length": "26", "date": "Fri, 21 Mar 2025 02:10:26 GMT"} })
63002025-03-21T02:10:27.326ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
63012025-03-21T02:10:27.327ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
63022025-03-21T02:10:27.327ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
63032025-03-21T02:10:27.328ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "971aa3a0-79b3-45f5-93e9-d67ead9547bc", "content-length": "26", "date": "Fri, 21 Mar 2025 02:10:26 GMT"} })
63042025-03-21T02:10:27.328ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
63052025-03-21T02:10:27.328ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
63062025-03-21T02:10:27.329ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
63072025-03-21T02:10:27.334ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
63082025-03-21T02:10:27.334ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
63092025-03-21T02:10:27.335ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "e238c6ec-0f3b-49ce-abe4-1516ef3d1a64", "date": "Fri, 21 Mar 2025 02:10:27 GMT"} })
63102025-03-21T02:10:27.335ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63112025-03-21T02:10:27.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63122025-03-21T02:10:27.337ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63132025-03-21T02:10:27.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63142025-03-21T02:10:27.341ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63152025-03-21T02:10:27.341ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63162025-03-21T02:10:27.344ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
63172025-03-21T02:10:27.344ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
63182025-03-21T02:10:27.344ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "902793bc-1f74-41a2-b9e3-dd581e17e2f7", "date": "Fri, 21 Mar 2025 02:10:27 GMT"} })
63192025-03-21T02:10:27.346ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 427.162396ms iteration = 13 status = {}
63202025-03-21T02:10:27.352ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
63212025-03-21T02:10:27.352ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
63222025-03-21T02:10:27.352ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfb3d1c7-faa5-452d-8f00-b0d5f4ed085a", "content-length": "26", "date": "Fri, 21 Mar 2025 02:10:26 GMT"} })
63232025-03-21T02:10:27.352ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
63242025-03-21T02:10:27.355ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
63252025-03-21T02:10:27.410ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
63262025-03-21T02:10:27.410ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "8654fd35-f654-40be-85b1-b5ef78b884a5", "date": "Fri, 21 Mar 2025 02:10:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
63272025-03-21T02:10:27.410ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
63282025-03-21T02:10:27.412ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "e583cab2-edfd-4a5b-b502-3187333fcdc4", "date": "Fri, 21 Mar 2025 02:10:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
63292025-03-21T02:10:27.412ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
63302025-03-21T02:10:27.414ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
63312025-03-21T02:10:27.414ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 494.638421ms iteration = 9 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
63322025-03-21T02:10:27.455ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 536.030081ms iteration = 13 status = {}
63332025-03-21T02:10:27.480ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
63342025-03-21T02:10:27.480ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "e29b44c2-55d4-49e3-a992-6fb0c10d13a3", "content-length": "1", "date": "Fri, 21 Mar 2025 02:10:27 GMT"} })
63352025-03-21T02:10:27.481ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
63362025-03-21T02:10:27.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
63372025-03-21T02:10:27.483ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 564.327404ms iteration = 13 status = null
63382025-03-21T02:10:27.510ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
63392025-03-21T02:10:27.625ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
63402025-03-21T02:10:27.626ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "fec54f14-abb9-4cf9-8183-e8671c58ce95", "date": "Fri, 21 Mar 2025 02:10:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
63412025-03-21T02:10:27.626ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
63422025-03-21T02:10:27.628ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "ec25120e-c6d8-4473-a3fe-b9c711f1c598", "date": "Fri, 21 Mar 2025 02:10:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
63432025-03-21T02:10:27.628ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
63442025-03-21T02:10:27.628ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "6640e877-3642-4941-bc7c-544ec667b75b", "date": "Fri, 21 Mar 2025 02:10:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
63452025-03-21T02:10:27.628ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
63462025-03-21T02:10:27.630ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
63472025-03-21T02:10:27.630ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 711.323754ms iteration = 9 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
63482025-03-21T02:10:27.658ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63492025-03-21T02:10:27.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63502025-03-21T02:10:27.661ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
63512025-03-21T02:10:27.661ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63522025-03-21T02:10:27.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63532025-03-21T02:10:27.672ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
63542025-03-21T02:10:27.672ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7d7cfc9-0ec0-4bcb-a7ca-c50bdedeb00d", "content-length": "134", "date": "Fri, 21 Mar 2025 02:10:27 GMT"} })
63552025-03-21T02:10:27.672ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
63562025-03-21T02:10:27.672ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
63572025-03-21T02:10:27.673ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
63582025-03-21T02:10:27.673ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
63592025-03-21T02:10:27.673ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
63602025-03-21T02:10:27.673ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
63612025-03-21T02:10:27.673ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
63622025-03-21T02:10:27.673ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
63632025-03-21T02:10:27.673ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
63642025-03-21T02:10:27.673ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
63652025-03-21T02:10:27.673ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
63662025-03-21T02:10:27.673ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63672025-03-21T02:10:27.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63682025-03-21T02:10:27.675ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
63692025-03-21T02:10:27.675ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad400edd-77ed-4d13-a05c-f062fd6f66c3", "content-length": "2", "date": "Fri, 21 Mar 2025 02:10:27 GMT"} })
63702025-03-21T02:10:27.675ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
63712025-03-21T02:10:27.676ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c45705e-f530-4d23-9759-4bbfd658a141", "content-length": "52", "date": "Fri, 21 Mar 2025 02:10:27 GMT"} })
63722025-03-21T02:10:27.676ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
63732025-03-21T02:10:27.676ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
63742025-03-21T02:10:27.718ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
63752025-03-21T02:10:27.718ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
63762025-03-21T02:10:27.748ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
63772025-03-21T02:10:27.748ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:51380
63782025-03-21T02:10:27.748ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:51380 req_id = bfd8acb2-3c04-431d-aec1-2425e76d3c63 roles = RoleSet { roles: {} } uri = /metrics/producers
63792025-03-21T02:10:27.749ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:51380 req_id = bfd8acb2-3c04-431d-aec1-2425e76d3c63 resource = Database result = Ok(()) uri = /metrics/producers
63802025-03-21T02:10:27.775ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "dfed311d-6190-4272-8631-f5d61fb99246", "date": "Fri, 21 Mar 2025 02:10:26 GMT"} })
63812025-03-21T02:10:27.775ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
63822025-03-21T02:10:27.775ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
63832025-03-21T02:10:27.775ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a67ae07-6f0e-452b-ad21-8acece2a8826", "content-length": "682", "date": "Fri, 21 Mar 2025 02:10:26 GMT"} })
63842025-03-21T02:10:27.775ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63852025-03-21T02:10:27.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63862025-03-21T02:10:27.779ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63872025-03-21T02:10:27.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63882025-03-21T02:10:27.784ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63892025-03-21T02:10:27.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63902025-03-21T02:10:27.786ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63912025-03-21T02:10:27.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63922025-03-21T02:10:27.788ZINFOnexus: assigned collector to new producer collector_id = a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53 file = nexus/src/app/oximeter.rs:76 producer_id = e05eb799-8cf0-4702-bd3b-9994ae86118a
63932025-03-21T02:10:27.788ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 39340 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:51380 req_id = bfd8acb2-3c04-431d-aec1-2425e76d3c63 response_code = 201 uri = /metrics/producers
63942025-03-21T02:10:27.788ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "bfd8acb2-3c04-431d-aec1-2425e76d3c63", "content-length": "41", "date": "Fri, 21 Mar 2025 02:10:27 GMT"} })
63952025-03-21T02:10:27.789ZDEBGnexus (producer-registration-task): registered with nexus successfully
63962025-03-21T02:10:27.789ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
63972025-03-21T02:10:27.794ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 875.249527ms iteration = 13 status = {}
63982025-03-21T02:10:29.720ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 336 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 7965d8a7-0b00-4da5-8a31-e9e324fdb070 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
63992025-03-21T02:10:36.920ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 38 reason = Timeout
64002025-03-21T02:10:36.920ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
64012025-03-21T02:10:36.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64022025-03-21T02:10:36.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64032025-03-21T02:10:36.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64042025-03-21T02:10:36.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64052025-03-21T02:10:36.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64062025-03-21T02:10:36.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64072025-03-21T02:10:36.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64082025-03-21T02:10:36.924ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
64092025-03-21T02:10:36.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64102025-03-21T02:10:36.924ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.468608ms iteration = 38 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
64112025-03-21T02:10:36.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64122025-03-21T02:10:36.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64132025-03-21T02:10:36.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64142025-03-21T02:10:36.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64152025-03-21T02:10:36.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
64162025-03-21T02:10:36.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64172025-03-21T02:10:36.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64182025-03-21T02:10:36.939ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
64192025-03-21T02:10:36.939ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.430038ms iteration = 38 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
64202025-03-21T02:10:39.720ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 304 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 7275b579-8397-4448-9078-0cfaa839852b response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
64212025-03-21T02:10:46.920ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 39 reason = Timeout
64222025-03-21T02:10:46.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64232025-03-21T02:10:46.920ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
64242025-03-21T02:10:46.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64252025-03-21T02:10:46.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64262025-03-21T02:10:46.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64272025-03-21T02:10:46.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64282025-03-21T02:10:46.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64292025-03-21T02:10:46.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64302025-03-21T02:10:46.923ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.576973ms iteration = 39 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
64312025-03-21T02:10:46.924ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
64322025-03-21T02:10:46.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64332025-03-21T02:10:46.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64342025-03-21T02:10:46.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64352025-03-21T02:10:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64362025-03-21T02:10:46.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64372025-03-21T02:10:46.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
64382025-03-21T02:10:46.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64392025-03-21T02:10:46.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64402025-03-21T02:10:46.937ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
64412025-03-21T02:10:46.937ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.321782ms iteration = 39 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
64422025-03-21T02:10:49.720ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 287 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = a4c97106-a74b-4acf-95cc-8252f8e4cf9a response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
64432025-03-21T02:10:54.711ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:49544
64442025-03-21T02:10:54.711ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49544 req_id = da96f715-29f8-432a-9492-f606e1d91cd1 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
64452025-03-21T02:10:54.712ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49544 req_id = da96f715-29f8-432a-9492-f606e1d91cd1 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
64462025-03-21T02:10:54.715ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3502 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49544 req_id = da96f715-29f8-432a-9492-f606e1d91cd1 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
64472025-03-21T02:10:54.715ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49544 req_id = 03a9852d-0b8d-470e-8f34-c1f224cee0b5 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
64482025-03-21T02:10:54.715ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49544 req_id = 03a9852d-0b8d-470e-8f34-c1f224cee0b5 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
64492025-03-21T02:10:54.718ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2645 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49544 req_id = 03a9852d-0b8d-470e-8f34-c1f224cee0b5 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
64502025-03-21T02:10:56.919ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 40 reason = Timeout
64512025-03-21T02:10:56.919ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64522025-03-21T02:10:56.919ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
64532025-03-21T02:10:56.919ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 14 reason = Timeout
64542025-03-21T02:10:56.919ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
64552025-03-21T02:10:56.919ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64562025-03-21T02:10:56.919ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 14 reason = Timeout
64572025-03-21T02:10:56.919ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64582025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 14 reason = Timeout
64592025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
64602025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 14 reason = Timeout
64612025-03-21T02:10:56.920ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
64622025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
64632025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 14 reason = Timeout
64642025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
64652025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
64662025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 14 reason = Timeout
64672025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
64682025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
64692025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
64702025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64712025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 14 reason = Timeout
64722025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
64732025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
64742025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 14 reason = Timeout
64752025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
64762025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64772025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
64782025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
64792025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
64802025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
64812025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 15 reason = Timeout
64822025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64832025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
64842025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
64852025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 14 reason = Timeout
64862025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
64872025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 14 reason = Timeout
64882025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64892025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64902025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64912025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
64922025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
64932025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64942025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64952025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
64962025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
64972025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
64982025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64992025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
65002025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65012025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
65022025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
65032025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65042025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
65052025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
65062025-03-21T02:10:56.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
65072025-03-21T02:10:56.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
65082025-03-21T02:10:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
65092025-03-21T02:10:56.923ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65102025-03-21T02:10:56.924ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.090861ms iteration = 14 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
65112025-03-21T02:10:56.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65122025-03-21T02:10:56.924ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65132025-03-21T02:10:56.924ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
65142025-03-21T02:10:56.925ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
65152025-03-21T02:10:56.925ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
65162025-03-21T02:10:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
65172025-03-21T02:10:56.925ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
65182025-03-21T02:10:56.925ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65192025-03-21T02:10:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65202025-03-21T02:10:56.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
65212025-03-21T02:10:56.927ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
65222025-03-21T02:10:56.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
65232025-03-21T02:10:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
65242025-03-21T02:10:56.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65252025-03-21T02:10:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65262025-03-21T02:10:56.928ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1365, tv_nsec: 550439864 } })
65272025-03-21T02:10:56.928ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
65282025-03-21T02:10:56.928ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65292025-03-21T02:10:56.930ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1365, tv_nsec: 552339598 } })
65302025-03-21T02:10:56.930ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
65312025-03-21T02:10:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
65322025-03-21T02:10:56.930ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
65332025-03-21T02:10:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
65342025-03-21T02:10:56.930ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 10.488022ms iteration = 14 status = {"errors":[],"garbage_collect_requested":[]}
65352025-03-21T02:10:56.930ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65362025-03-21T02:10:56.931ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65372025-03-21T02:10:56.932ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65382025-03-21T02:10:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
65392025-03-21T02:10:56.934ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65402025-03-21T02:10:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
65412025-03-21T02:10:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65422025-03-21T02:10:56.934ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
65432025-03-21T02:10:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
65442025-03-21T02:10:56.935ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
65452025-03-21T02:10:56.935ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65462025-03-21T02:10:56.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65472025-03-21T02:10:56.938ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65482025-03-21T02:10:56.940ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
65492025-03-21T02:10:56.940ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65502025-03-21T02:10:56.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65512025-03-21T02:10:56.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65522025-03-21T02:10:56.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65532025-03-21T02:10:56.942ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
65542025-03-21T02:10:56.942ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65552025-03-21T02:10:56.942ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
65562025-03-21T02:10:56.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
65572025-03-21T02:10:56.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65582025-03-21T02:10:56.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
65592025-03-21T02:10:56.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65602025-03-21T02:10:56.944ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1365, tv_nsec: 566509874 } })
65612025-03-21T02:10:56.944ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
65622025-03-21T02:10:56.944ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65632025-03-21T02:10:56.945ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65642025-03-21T02:10:56.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
65652025-03-21T02:10:56.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65662025-03-21T02:10:56.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
65672025-03-21T02:10:56.947ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
65682025-03-21T02:10:56.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65692025-03-21T02:10:56.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
65702025-03-21T02:10:56.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
65712025-03-21T02:10:56.948ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
65722025-03-21T02:10:56.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
65732025-03-21T02:10:56.964ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
65742025-03-21T02:10:56.964ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 44.674328ms iteration = 40 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
65752025-03-21T02:10:56.978ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65762025-03-21T02:10:56.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65772025-03-21T02:10:56.979ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 59.215351ms iteration = 14 status = {"errors":[],"requests_created_ok":[]}
65782025-03-21T02:10:56.980ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
65792025-03-21T02:10:56.981ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65802025-03-21T02:10:56.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
65812025-03-21T02:10:56.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65822025-03-21T02:10:56.981ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65832025-03-21T02:10:56.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65842025-03-21T02:10:56.982ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 62.242884ms iteration = 14 status = {"errors":[],"finish_invoked_ok":[]}
65852025-03-21T02:10:56.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65862025-03-21T02:10:56.985ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
65872025-03-21T02:10:56.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
65882025-03-21T02:10:56.986ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
65892025-03-21T02:10:56.986ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65902025-03-21T02:10:56.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65912025-03-21T02:10:56.986ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
65922025-03-21T02:10:56.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
65932025-03-21T02:10:56.987ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
65942025-03-21T02:10:56.987ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 67.019837ms iteration = 14 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
65952025-03-21T02:10:56.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 67.378243ms iteration = 40 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
65962025-03-21T02:10:56.987ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
65972025-03-21T02:10:56.987ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 67.619627ms iteration = 14 status = 0
65982025-03-21T02:10:56.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
65992025-03-21T02:10:56.992ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
66002025-03-21T02:10:56.992ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
66012025-03-21T02:10:56.992ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
66022025-03-21T02:10:56.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
66032025-03-21T02:10:56.995ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
66042025-03-21T02:10:56.995ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 75.867971ms iteration = 14 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
66052025-03-21T02:10:56.997ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
66062025-03-21T02:10:56.997ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
66072025-03-21T02:10:56.998ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "816e5a83-c3f4-427f-a376-b9fd9efe961a", "content-length": "26", "date": "Fri, 21 Mar 2025 02:10:56 GMT"} })
66082025-03-21T02:10:56.998ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
66092025-03-21T02:10:57.000ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
66102025-03-21T02:10:57.011ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
66112025-03-21T02:10:57.011ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
66122025-03-21T02:10:57.011ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "862226c7-12b3-4909-a94b-c617f3047a74", "content-length": "26", "date": "Fri, 21 Mar 2025 02:10:56 GMT"} })
66132025-03-21T02:10:57.011ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
66142025-03-21T02:10:57.013ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
66152025-03-21T02:10:57.023ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 103.573563ms iteration = 14 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
66162025-03-21T02:10:57.025ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 105.193932ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
66172025-03-21T02:10:57.027ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 107.42418ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
66182025-03-21T02:10:57.031ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
66192025-03-21T02:10:57.031ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
66202025-03-21T02:10:57.031ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 111.806617ms iteration = 14 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
66212025-03-21T02:10:57.033ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
66222025-03-21T02:10:57.033ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
66232025-03-21T02:10:57.033ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "21dc779a-83e5-4336-9ccf-ec24fab2d8b2", "content-length": "26", "date": "Fri, 21 Mar 2025 02:10:56 GMT"} })
66242025-03-21T02:10:57.033ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
66252025-03-21T02:10:57.034ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
66262025-03-21T02:10:57.061ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 141.394762ms iteration = 15 status = {"physical_disks_added":0}
66272025-03-21T02:10:57.087ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
66282025-03-21T02:10:57.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
66292025-03-21T02:10:57.089ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66302025-03-21T02:10:57.091ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
66312025-03-21T02:10:57.091ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
66322025-03-21T02:10:57.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
66332025-03-21T02:10:57.092ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66342025-03-21T02:10:57.094ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
66352025-03-21T02:10:57.094ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
66362025-03-21T02:10:57.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
66372025-03-21T02:10:57.096ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 176.115406ms iteration = 14 status = {}
66382025-03-21T02:10:57.103ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
66392025-03-21T02:10:57.104ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f43a1d4-1960-4b72-87ad-f4f7c2286097", "content-length": "1", "date": "Fri, 21 Mar 2025 02:10:56 GMT"} })
66402025-03-21T02:10:57.105ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
66412025-03-21T02:10:57.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
66422025-03-21T02:10:57.107ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 187.381202ms iteration = 14 status = null
66432025-03-21T02:10:57.114ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
66442025-03-21T02:10:57.114ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
66452025-03-21T02:10:57.175ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "eae5f015-235d-4c60-8aa2-f39aa0b66253", "content-length": "372", "date": "Fri, 21 Mar 2025 02:10:57 GMT"} })
66462025-03-21T02:10:57.175ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66472025-03-21T02:10:57.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66482025-03-21T02:10:57.190ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66492025-03-21T02:10:57.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66502025-03-21T02:10:57.192ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
66512025-03-21T02:10:57.192ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66522025-03-21T02:10:57.192ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66532025-03-21T02:10:57.193ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
66542025-03-21T02:10:57.193ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ca0c479-7398-456c-acff-60b0b6ec5950", "content-length": "691", "date": "Fri, 21 Mar 2025 02:10:56 GMT"} })
66552025-03-21T02:10:57.193ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
66562025-03-21T02:10:57.193ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
66572025-03-21T02:10:57.193ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
66582025-03-21T02:10:57.194ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 274.316145ms iteration = 14 status = {}
66592025-03-21T02:10:57.203ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
66602025-03-21T02:10:57.203ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "cace5b49-595b-46c9-940a-a9d2f6b9ebe4", "content-length": "134", "date": "Fri, 21 Mar 2025 02:10:56 GMT"} })
66612025-03-21T02:10:57.203ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
66622025-03-21T02:10:57.203ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
66632025-03-21T02:10:57.203ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
66642025-03-21T02:10:57.203ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
66652025-03-21T02:10:57.203ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66662025-03-21T02:10:57.203ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66672025-03-21T02:10:57.203ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66682025-03-21T02:10:57.203ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
66692025-03-21T02:10:57.204ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66702025-03-21T02:10:57.204ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
66712025-03-21T02:10:57.204ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
66722025-03-21T02:10:57.204ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66732025-03-21T02:10:57.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66742025-03-21T02:10:57.206ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
66752025-03-21T02:10:57.206ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "73ef1180-5e14-4a1a-9173-aadf9b2e5f1e", "content-length": "2", "date": "Fri, 21 Mar 2025 02:10:56 GMT"} })
66762025-03-21T02:10:57.206ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
66772025-03-21T02:10:57.206ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "e241e008-8504-4a94-bf0d-1672d5aa1ce0", "content-length": "52", "date": "Fri, 21 Mar 2025 02:10:56 GMT"} })
66782025-03-21T02:10:57.206ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
66792025-03-21T02:10:57.206ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
66802025-03-21T02:10:57.216ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
66812025-03-21T02:10:57.216ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
66822025-03-21T02:10:57.293ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "a57a593d-5a18-474c-85df-5592336cc26f", "date": "Fri, 21 Mar 2025 02:10:56 GMT"} })
66832025-03-21T02:10:57.293ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
66842025-03-21T02:10:57.293ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
66852025-03-21T02:10:57.293ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "8ed2eb77-88ee-48ca-b3b8-700473d2e53a", "date": "Fri, 21 Mar 2025 02:10:56 GMT"} })
66862025-03-21T02:10:57.293ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66872025-03-21T02:10:57.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66882025-03-21T02:10:57.293ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "eeb4f357-c35f-454f-9e17-c61e3635b01f", "content-length": "682", "date": "Fri, 21 Mar 2025 02:10:56 GMT"} })
66892025-03-21T02:10:57.293ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66902025-03-21T02:10:57.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66912025-03-21T02:10:57.296ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66922025-03-21T02:10:57.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66932025-03-21T02:10:57.297ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66942025-03-21T02:10:57.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66952025-03-21T02:10:57.299ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66962025-03-21T02:10:57.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66972025-03-21T02:10:57.301ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66982025-03-21T02:10:57.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66992025-03-21T02:10:57.303ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
67002025-03-21T02:10:57.303ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
67012025-03-21T02:10:57.303ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
67022025-03-21T02:10:57.303ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "a39c9afc-93de-4afd-8296-dacb85642c39", "date": "Fri, 21 Mar 2025 02:10:57 GMT"} })
67032025-03-21T02:10:57.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
67042025-03-21T02:10:57.305ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 385.43097ms iteration = 14 status = {}
67052025-03-21T02:10:57.310ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 390.959186ms iteration = 14 status = {}
67062025-03-21T02:10:59.720ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 285 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 80a1e448-4580-4c97-a86e-6ebdd9c5cddc response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
67072025-03-21T02:11:06.920ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 41 reason = Timeout
67082025-03-21T02:11:06.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67092025-03-21T02:11:06.920ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
67102025-03-21T02:11:06.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67112025-03-21T02:11:06.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67122025-03-21T02:11:06.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67132025-03-21T02:11:06.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67142025-03-21T02:11:06.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67152025-03-21T02:11:06.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67162025-03-21T02:11:06.924ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.962569ms iteration = 41 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
67172025-03-21T02:11:06.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
67182025-03-21T02:11:06.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67192025-03-21T02:11:06.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67202025-03-21T02:11:06.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67212025-03-21T02:11:06.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67222025-03-21T02:11:06.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67232025-03-21T02:11:06.932ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
67242025-03-21T02:11:06.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67252025-03-21T02:11:06.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67262025-03-21T02:11:06.939ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
67272025-03-21T02:11:06.939ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.716305ms iteration = 41 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
67282025-03-21T02:11:09.720ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 290 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = c08fd1d2-6789-4a46-9aa9-312ff79624df response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
67292025-03-21T02:11:16.919ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 42 reason = Timeout
67302025-03-21T02:11:16.919ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67312025-03-21T02:11:16.919ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
67322025-03-21T02:11:16.919ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67332025-03-21T02:11:16.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67342025-03-21T02:11:16.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67352025-03-21T02:11:16.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67362025-03-21T02:11:16.924ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
67372025-03-21T02:11:16.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67382025-03-21T02:11:16.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67392025-03-21T02:11:16.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67402025-03-21T02:11:16.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67412025-03-21T02:11:16.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67422025-03-21T02:11:16.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
67432025-03-21T02:11:16.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67442025-03-21T02:11:16.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67452025-03-21T02:11:16.938ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
67462025-03-21T02:11:16.938ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.030862ms iteration = 42 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
67472025-03-21T02:11:16.949ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67482025-03-21T02:11:16.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67492025-03-21T02:11:16.979ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 60.239918ms iteration = 42 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
67502025-03-21T02:11:19.720ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 260 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = e6dafc8e-0047-4004-87e5-63ea431a262c response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
67512025-03-21T02:11:26.919ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 43 reason = Timeout
67522025-03-21T02:11:26.919ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
67532025-03-21T02:11:26.919ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67542025-03-21T02:11:26.919ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
67552025-03-21T02:11:26.919ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
67562025-03-21T02:11:26.919ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 15 reason = Timeout
67572025-03-21T02:11:26.919ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
67582025-03-21T02:11:26.919ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
67592025-03-21T02:11:26.919ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 22.83\u{b5}s iteration = 9 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
67602025-03-21T02:11:26.919ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
67612025-03-21T02:11:26.919ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
67622025-03-21T02:11:26.919ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 15 reason = Timeout
67632025-03-21T02:11:26.919ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
67642025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67652025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
67662025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
67672025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 16 reason = Timeout
67682025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 8 reason = Timeout
67692025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
67702025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67712025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
67722025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 15 reason = Timeout
67732025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
67742025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
67752025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 8 reason = Timeout
67762025-03-21T02:11:26.920ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
67772025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 8 reason = Timeout
67782025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
67792025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
67802025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
67812025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
67822025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
67832025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
67842025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
67852025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
67862025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
67872025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
67882025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 15 reason = Timeout
67892025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
67902025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 15 reason = Timeout
67912025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
67922025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
67932025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
67942025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
67952025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 15 reason = Timeout
67962025-03-21T02:11:26.920ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
67972025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
67982025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
67992025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
68002025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 15 reason = Timeout
68012025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
68022025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
68032025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
68042025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
68052025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
68062025-03-21T02:11:26.920ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 02:01:26.920245146 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
68072025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
68082025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 15 reason = Timeout
68092025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
68102025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 15 reason = Timeout
68112025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 15 reason = Timeout
68122025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
68132025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
68142025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
68152025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 8 reason = Timeout
68162025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
68172025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
68182025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
68192025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
68202025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68212025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
68222025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
68232025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
68242025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68252025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
68262025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
68272025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
68282025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
68292025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68302025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
68312025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
68322025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68332025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
68342025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
68352025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
68362025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68372025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
68382025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
68392025-03-21T02:11:26.920ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1395, tv_nsec: 542707733 } })
68402025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
68412025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68422025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
68432025-03-21T02:11:26.920ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1395, tv_nsec: 542746004 } })
68442025-03-21T02:11:26.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68452025-03-21T02:11:26.921ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
68462025-03-21T02:11:26.921ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.501806ms iteration = 8 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
68472025-03-21T02:11:26.921ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
68482025-03-21T02:11:26.921ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.508927ms iteration = 8 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
68492025-03-21T02:11:26.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
68502025-03-21T02:11:26.923ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.313438ms iteration = 15 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
68512025-03-21T02:11:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
68522025-03-21T02:11:26.923ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1395, tv_nsec: 545465341 } })
68532025-03-21T02:11:26.923ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
68542025-03-21T02:11:26.925ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68552025-03-21T02:11:26.925ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68562025-03-21T02:11:26.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68572025-03-21T02:11:26.927ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 7.331307ms iteration = 8 status = {"errors":[],"found_port_ok":[]}
68582025-03-21T02:11:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
68592025-03-21T02:11:26.928ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
68602025-03-21T02:11:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
68612025-03-21T02:11:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
68622025-03-21T02:11:26.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68632025-03-21T02:11:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68642025-03-21T02:11:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
68652025-03-21T02:11:26.930ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68662025-03-21T02:11:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68672025-03-21T02:11:26.931ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
68682025-03-21T02:11:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
68692025-03-21T02:11:26.931ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
68702025-03-21T02:11:26.932ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68712025-03-21T02:11:26.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
68722025-03-21T02:11:26.933ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68732025-03-21T02:11:26.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
68742025-03-21T02:11:26.933ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
68752025-03-21T02:11:26.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
68762025-03-21T02:11:26.934ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1395, tv_nsec: 556153307 } })
68772025-03-21T02:11:26.934ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
68782025-03-21T02:11:26.935ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
68792025-03-21T02:11:26.935ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68802025-03-21T02:11:26.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68812025-03-21T02:11:26.936ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68822025-03-21T02:11:26.936ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68832025-03-21T02:11:26.937ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 17.875451ms iteration = 15 status = {"errors":[],"finish_invoked_ok":[]}
68842025-03-21T02:11:26.938ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
68852025-03-21T02:11:26.938ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
68862025-03-21T02:11:26.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
68872025-03-21T02:11:26.939ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
68882025-03-21T02:11:26.939ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
68892025-03-21T02:11:26.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68902025-03-21T02:11:26.940ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68912025-03-21T02:11:26.941ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68922025-03-21T02:11:26.942ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
68932025-03-21T02:11:26.942ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68942025-03-21T02:11:26.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68952025-03-21T02:11:26.943ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 02:01:26.920245146 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
68962025-03-21T02:11:26.943ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 22.845718ms iteration = 8 status = {"expiration":"2025-03-21T02:01:26.920245146Z","pruned":[]}
68972025-03-21T02:11:26.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
68982025-03-21T02:11:26.944ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
68992025-03-21T02:11:26.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69002025-03-21T02:11:26.945ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 25.29304ms iteration = 43 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
69012025-03-21T02:11:26.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69022025-03-21T02:11:26.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69032025-03-21T02:11:26.946ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69042025-03-21T02:11:26.947ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69052025-03-21T02:11:26.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69062025-03-21T02:11:26.951ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69072025-03-21T02:11:26.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
69082025-03-21T02:11:26.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69092025-03-21T02:11:26.954ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
69102025-03-21T02:11:26.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
69112025-03-21T02:11:26.958ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
69122025-03-21T02:11:26.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
69132025-03-21T02:11:26.960ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69142025-03-21T02:11:26.960ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69152025-03-21T02:11:26.962ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
69162025-03-21T02:11:26.962ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69172025-03-21T02:11:26.962ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1395, tv_nsec: 584455760 } })
69182025-03-21T02:11:26.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69192025-03-21T02:11:26.963ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
69202025-03-21T02:11:26.964ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
69212025-03-21T02:11:26.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69222025-03-21T02:11:26.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69232025-03-21T02:11:26.968ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69242025-03-21T02:11:26.972ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69252025-03-21T02:11:26.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
69262025-03-21T02:11:26.972ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
69272025-03-21T02:11:26.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
69282025-03-21T02:11:26.975ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 55.300603ms iteration = 15 status = {"errors":[],"garbage_collect_requested":[]}
69292025-03-21T02:11:26.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
69302025-03-21T02:11:26.975ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
69312025-03-21T02:11:26.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
69322025-03-21T02:11:26.977ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69332025-03-21T02:11:26.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69342025-03-21T02:11:26.985ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 65.253556ms iteration = 15 status = {"errors":[],"requests_created_ok":[]}
69352025-03-21T02:11:26.987ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
69362025-03-21T02:11:26.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
69372025-03-21T02:11:26.994ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
69382025-03-21T02:11:26.994ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 74.299384ms iteration = 43 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
69392025-03-21T02:11:26.994ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
69402025-03-21T02:11:26.994ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
69412025-03-21T02:11:26.994ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
69422025-03-21T02:11:26.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
69432025-03-21T02:11:26.996ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69442025-03-21T02:11:26.997ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
69452025-03-21T02:11:26.997ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
69462025-03-21T02:11:26.997ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 76.811327ms iteration = 8 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
69472025-03-21T02:11:27.000ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
69482025-03-21T02:11:27.007ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
69492025-03-21T02:11:27.007ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 87.189718ms iteration = 15 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
69502025-03-21T02:11:27.008ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
69512025-03-21T02:11:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
69522025-03-21T02:11:27.009ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
69532025-03-21T02:11:27.009ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
69542025-03-21T02:11:27.009ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 89.320764ms iteration = 8 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
69552025-03-21T02:11:27.009ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
69562025-03-21T02:11:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
69572025-03-21T02:11:27.013ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69582025-03-21T02:11:27.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69592025-03-21T02:11:27.015ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
69602025-03-21T02:11:27.015ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
69612025-03-21T02:11:27.015ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
69622025-03-21T02:11:27.015ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
69632025-03-21T02:11:27.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
69642025-03-21T02:11:27.017ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
69652025-03-21T02:11:27.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
69662025-03-21T02:11:27.044ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
69672025-03-21T02:11:27.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
69682025-03-21T02:11:27.049ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
69692025-03-21T02:11:27.050ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
69702025-03-21T02:11:27.050ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
69712025-03-21T02:11:27.050ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
69722025-03-21T02:11:27.050ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
69732025-03-21T02:11:27.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
69742025-03-21T02:11:27.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
69752025-03-21T02:11:27.052ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 132.979105ms iteration = 16 status = {"physical_disks_added":0}
69762025-03-21T02:11:27.077ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
69772025-03-21T02:11:27.107ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
69782025-03-21T02:11:27.107ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
69792025-03-21T02:11:27.107ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fff396ff-37c3-4975-a647-a05ab07176e9", "content-length": "26", "date": "Fri, 21 Mar 2025 02:11:27 GMT"} })
69802025-03-21T02:11:27.108ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
69812025-03-21T02:11:27.109ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
69822025-03-21T02:11:27.123ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
69832025-03-21T02:11:27.123ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
69842025-03-21T02:11:27.123ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0b1b6ba-5099-405a-808a-a7c69e234354", "content-length": "372", "date": "Fri, 21 Mar 2025 02:11:26 GMT"} })
69852025-03-21T02:11:27.123ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69862025-03-21T02:11:27.124ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69872025-03-21T02:11:27.125ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69882025-03-21T02:11:27.126ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 206.168079ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
69892025-03-21T02:11:27.127ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
69902025-03-21T02:11:27.127ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69912025-03-21T02:11:27.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69922025-03-21T02:11:27.129ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
69932025-03-21T02:11:27.129ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
69942025-03-21T02:11:27.130ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69952025-03-21T02:11:27.130ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69962025-03-21T02:11:27.131ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69972025-03-21T02:11:27.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69982025-03-21T02:11:27.131ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69992025-03-21T02:11:27.132ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
70002025-03-21T02:11:27.132ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
70012025-03-21T02:11:27.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
70022025-03-21T02:11:27.134ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
70032025-03-21T02:11:27.134ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
70042025-03-21T02:11:27.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
70052025-03-21T02:11:27.134ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
70062025-03-21T02:11:27.134ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
70072025-03-21T02:11:27.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
70082025-03-21T02:11:27.138ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70092025-03-21T02:11:27.139ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70102025-03-21T02:11:27.140ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 220.132233ms iteration = 15 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
70112025-03-21T02:11:27.140ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
70122025-03-21T02:11:27.140ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70132025-03-21T02:11:27.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70142025-03-21T02:11:27.142ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
70152025-03-21T02:11:27.142ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
70162025-03-21T02:11:27.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
70172025-03-21T02:11:27.145ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
70182025-03-21T02:11:27.145ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
70192025-03-21T02:11:27.145ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
70202025-03-21T02:11:27.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
70212025-03-21T02:11:27.148ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 228.182853ms iteration = 15 status = 0
70222025-03-21T02:11:27.162ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
70232025-03-21T02:11:27.162ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
70242025-03-21T02:11:27.163ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd13cadc-72bb-4b1a-ab5c-432f52997922", "content-length": "26", "date": "Fri, 21 Mar 2025 02:11:27 GMT"} })
70252025-03-21T02:11:27.163ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
70262025-03-21T02:11:27.164ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
70272025-03-21T02:11:27.169ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
70282025-03-21T02:11:27.169ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
70292025-03-21T02:11:27.169ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
70302025-03-21T02:11:27.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
70312025-03-21T02:11:27.174ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 254.270167ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
70322025-03-21T02:11:27.180ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
70332025-03-21T02:11:27.180ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 260.267151ms iteration = 15 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
70342025-03-21T02:11:27.180ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
70352025-03-21T02:11:27.190ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
70362025-03-21T02:11:27.190ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
70372025-03-21T02:11:27.190ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
70382025-03-21T02:11:27.190ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
70392025-03-21T02:11:27.191ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0bb1cda-8eb2-446c-a798-650f430d6537", "content-length": "26", "date": "Fri, 21 Mar 2025 02:11:27 GMT"} })
70402025-03-21T02:11:27.191ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
70412025-03-21T02:11:27.193ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
70422025-03-21T02:11:27.193ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
70432025-03-21T02:11:27.193ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
70442025-03-21T02:11:27.193ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
70452025-03-21T02:11:27.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
70462025-03-21T02:11:27.195ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70472025-03-21T02:11:27.197ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70482025-03-21T02:11:27.198ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70492025-03-21T02:11:27.198ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
70502025-03-21T02:11:27.198ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
70512025-03-21T02:11:27.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
70522025-03-21T02:11:27.199ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
70532025-03-21T02:11:27.199ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
70542025-03-21T02:11:27.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
70552025-03-21T02:11:27.201ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
70562025-03-21T02:11:27.201ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
70572025-03-21T02:11:27.201ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-21 02:04:30.886360 UTC dns_group = internal generation = 1
70582025-03-21T02:11:27.201ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 281.701644ms iteration = 8 status = {"generation":1}
70592025-03-21T02:11:27.202ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
70602025-03-21T02:11:27.202ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
70612025-03-21T02:11:27.202ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
70622025-03-21T02:11:27.204ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
70632025-03-21T02:11:27.204ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
70642025-03-21T02:11:27.204ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-21 02:04:31.662541 UTC dns_group = external generation = 2
70652025-03-21T02:11:27.204ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 284.214448ms iteration = 8 status = {"generation":2}
70662025-03-21T02:11:27.212ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
70672025-03-21T02:11:27.212ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 292.533123ms iteration = 8 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
70682025-03-21T02:11:27.215ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
70692025-03-21T02:11:27.215ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
70702025-03-21T02:11:27.215ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 295.441303ms iteration = 15 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
70712025-03-21T02:11:27.233ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
70722025-03-21T02:11:27.238ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) }
70732025-03-21T02:11:27.238ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 318.22194ms iteration = 8 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
70742025-03-21T02:11:27.255ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 335.125464ms iteration = 15 status = {}
70752025-03-21T02:11:27.267ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "9f1f38ac-c6ba-4230-a647-83bf71a8cc21", "date": "Fri, 21 Mar 2025 02:11:27 GMT"} })
70762025-03-21T02:11:27.267ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70772025-03-21T02:11:27.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70782025-03-21T02:11:27.270ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70792025-03-21T02:11:27.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70802025-03-21T02:11:27.273ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70812025-03-21T02:11:27.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70822025-03-21T02:11:27.309ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
70832025-03-21T02:11:27.309ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "50572380-ad0d-4800-80b4-22d4681d9b78", "date": "Fri, 21 Mar 2025 02:11:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70842025-03-21T02:11:27.309ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70852025-03-21T02:11:27.310ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "147c7c1b-f41f-45ed-952a-9ff2b6a59e8f", "date": "Fri, 21 Mar 2025 02:11:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70862025-03-21T02:11:27.310ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70872025-03-21T02:11:27.312ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70882025-03-21T02:11:27.312ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 392.239049ms iteration = 10 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
70892025-03-21T02:11:27.313ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
70902025-03-21T02:11:27.314ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "48f07ae3-8087-476c-a416-e1c827e65e63", "content-length": "1", "date": "Fri, 21 Mar 2025 02:11:26 GMT"} })
70912025-03-21T02:11:27.315ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
70922025-03-21T02:11:27.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
70932025-03-21T02:11:27.333ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
70942025-03-21T02:11:27.333ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3f15d4f-645c-4e0c-938b-2ec2c24fc6d7", "content-length": "691", "date": "Fri, 21 Mar 2025 02:11:27 GMT"} })
70952025-03-21T02:11:27.334ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
70962025-03-21T02:11:27.334ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
70972025-03-21T02:11:27.334ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
70982025-03-21T02:11:27.334ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 414.668569ms iteration = 15 status = {}
70992025-03-21T02:11:27.335ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 415.983052ms iteration = 15 status = null
71002025-03-21T02:11:27.336ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
71012025-03-21T02:11:27.336ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
71022025-03-21T02:11:27.337ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "36f036f7-66a8-4ae6-a0b9-892159860864", "date": "Fri, 21 Mar 2025 02:11:27 GMT"} })
71032025-03-21T02:11:27.338ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
71042025-03-21T02:11:27.339ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 418.908162ms iteration = 15 status = {}
71052025-03-21T02:11:27.339ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "9f3c7adc-bed5-4c56-9439-45456fad3503", "date": "Fri, 21 Mar 2025 02:11:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
71062025-03-21T02:11:27.339ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
71072025-03-21T02:11:27.340ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "983f97a1-985c-44db-b312-24c3650e730b", "date": "Fri, 21 Mar 2025 02:11:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
71082025-03-21T02:11:27.340ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
71092025-03-21T02:11:27.342ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "48d1efa4-270e-4042-b340-2e48155d5c17", "date": "Fri, 21 Mar 2025 02:11:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
71102025-03-21T02:11:27.342ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
71112025-03-21T02:11:27.343ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
71122025-03-21T02:11:27.343ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 423.35371ms iteration = 10 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
71132025-03-21T02:11:27.386ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71142025-03-21T02:11:27.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71152025-03-21T02:11:27.389ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
71162025-03-21T02:11:27.389ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71172025-03-21T02:11:27.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71182025-03-21T02:11:27.399ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
71192025-03-21T02:11:27.399ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "21ff53ec-840d-45ec-9d5e-f541cf577fd7", "content-length": "134", "date": "Fri, 21 Mar 2025 02:11:26 GMT"} })
71202025-03-21T02:11:27.399ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
71212025-03-21T02:11:27.399ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
71222025-03-21T02:11:27.400ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
71232025-03-21T02:11:27.400ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
71242025-03-21T02:11:27.400ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
71252025-03-21T02:11:27.400ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
71262025-03-21T02:11:27.400ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
71272025-03-21T02:11:27.400ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
71282025-03-21T02:11:27.400ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
71292025-03-21T02:11:27.400ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
71302025-03-21T02:11:27.400ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
71312025-03-21T02:11:27.400ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71322025-03-21T02:11:27.400ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71332025-03-21T02:11:27.402ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
71342025-03-21T02:11:27.402ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9661672-cd13-4510-95eb-10e79e66d628", "content-length": "2", "date": "Fri, 21 Mar 2025 02:11:27 GMT"} })
71352025-03-21T02:11:27.402ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
71362025-03-21T02:11:27.402ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "2db3b33e-a6e3-427d-ac32-7311a3ba5434", "content-length": "52", "date": "Fri, 21 Mar 2025 02:11:26 GMT"} })
71372025-03-21T02:11:27.402ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
71382025-03-21T02:11:27.402ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
71392025-03-21T02:11:27.489ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "1f524b22-dc67-4837-bf48-2c56e1b769e1", "date": "Fri, 21 Mar 2025 02:11:27 GMT"} })
71402025-03-21T02:11:27.489ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
71412025-03-21T02:11:27.489ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
71422025-03-21T02:11:27.489ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f30b78a-fbac-4a4d-8d0a-0c4904252cb6", "content-length": "682", "date": "Fri, 21 Mar 2025 02:11:27 GMT"} })
71432025-03-21T02:11:27.489ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71442025-03-21T02:11:27.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71452025-03-21T02:11:27.493ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71462025-03-21T02:11:27.493ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71472025-03-21T02:11:27.497ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71482025-03-21T02:11:27.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71492025-03-21T02:11:27.499ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71502025-03-21T02:11:27.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71512025-03-21T02:11:27.506ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 586.662353ms iteration = 15 status = {}
71522025-03-21T02:11:29.720ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 274 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 29d888dd-3429-412c-8832-d306a60d081b response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
71532025-03-21T02:11:36.920ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
71542025-03-21T02:11:36.920ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 44 reason = Timeout
71552025-03-21T02:11:36.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71562025-03-21T02:11:36.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71572025-03-21T02:11:36.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71582025-03-21T02:11:36.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71592025-03-21T02:11:36.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71602025-03-21T02:11:36.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
71612025-03-21T02:11:36.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71622025-03-21T02:11:36.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71632025-03-21T02:11:36.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71642025-03-21T02:11:36.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71652025-03-21T02:11:36.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71662025-03-21T02:11:36.931ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
71672025-03-21T02:11:36.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71682025-03-21T02:11:36.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71692025-03-21T02:11:36.940ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
71702025-03-21T02:11:36.940ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.987988ms iteration = 44 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
71712025-03-21T02:11:36.951ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71722025-03-21T02:11:36.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71732025-03-21T02:11:36.981ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 60.376481ms iteration = 44 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
71742025-03-21T02:11:39.720ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 7b8f5ac9-ce33-4226-ae95-c95ab2570595 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
71752025-03-21T02:11:46.920ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 45 reason = Timeout
71762025-03-21T02:11:46.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71772025-03-21T02:11:46.920ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
71782025-03-21T02:11:46.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71792025-03-21T02:11:46.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71802025-03-21T02:11:46.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71812025-03-21T02:11:46.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71822025-03-21T02:11:46.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
71832025-03-21T02:11:46.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71842025-03-21T02:11:46.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71852025-03-21T02:11:46.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71862025-03-21T02:11:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71872025-03-21T02:11:46.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71882025-03-21T02:11:46.931ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
71892025-03-21T02:11:46.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71902025-03-21T02:11:46.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71912025-03-21T02:11:46.943ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
71922025-03-21T02:11:46.943ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.688155ms iteration = 45 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
71932025-03-21T02:11:46.953ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71942025-03-21T02:11:46.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71952025-03-21T02:11:46.983ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.842974ms iteration = 45 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
71962025-03-21T02:11:49.720ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 303 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 7f960b0a-d472-489d-a5e4-6f0101249e7f response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
71972025-03-21T02:11:54.712ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:60359
71982025-03-21T02:11:54.712ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60359 req_id = 7e04dbf5-0729-408b-b586-fb0602f615bb roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
71992025-03-21T02:11:54.712ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60359 req_id = 7e04dbf5-0729-408b-b586-fb0602f615bb resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
72002025-03-21T02:11:54.715ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3055 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60359 req_id = 7e04dbf5-0729-408b-b586-fb0602f615bb response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
72012025-03-21T02:11:54.715ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60359 req_id = 0a452f29-0440-4a9a-8cee-bc5e3f574058 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
72022025-03-21T02:11:54.715ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60359 req_id = 0a452f29-0440-4a9a-8cee-bc5e3f574058 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
72032025-03-21T02:11:54.717ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2475 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60359 req_id = 0a452f29-0440-4a9a-8cee-bc5e3f574058 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
72042025-03-21T02:11:56.920ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 46 reason = Timeout
72052025-03-21T02:11:56.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
72062025-03-21T02:11:56.920ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
72072025-03-21T02:11:56.920ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
72082025-03-21T02:11:56.920ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 16 reason = Timeout
72092025-03-21T02:11:56.920ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72102025-03-21T02:11:56.920ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
72112025-03-21T02:11:56.920ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 16 reason = Timeout
72122025-03-21T02:11:56.920ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
72132025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
72142025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 16 reason = Timeout
72152025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
72162025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72172025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 17 reason = Timeout
72182025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 16 reason = Timeout
72192025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
72202025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72212025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 16 reason = Timeout
72222025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
72232025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 16 reason = Timeout
72242025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
72252025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 16 reason = Timeout
72262025-03-21T02:11:56.921ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
72272025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72282025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 16 reason = Timeout
72292025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
72302025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 16 reason = Timeout
72312025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
72322025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 16 reason = Timeout
72332025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
72342025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
72352025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
72362025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
72372025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
72382025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
72392025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
72402025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
72412025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
72422025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
72432025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
72442025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72452025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
72462025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
72472025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
72482025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
72492025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72502025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72512025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
72522025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72532025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
72542025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
72552025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
72562025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
72572025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
72582025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72592025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
72602025-03-21T02:11:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
72612025-03-21T02:11:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
72622025-03-21T02:11:56.923ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.574495ms iteration = 16 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
72632025-03-21T02:11:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
72642025-03-21T02:11:56.923ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72652025-03-21T02:11:56.924ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72662025-03-21T02:11:56.926ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
72672025-03-21T02:11:56.926ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
72682025-03-21T02:11:56.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
72692025-03-21T02:11:56.926ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
72702025-03-21T02:11:56.926ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72712025-03-21T02:11:56.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72722025-03-21T02:11:56.932ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
72732025-03-21T02:11:56.932ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
72742025-03-21T02:11:56.932ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
72752025-03-21T02:11:56.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
72762025-03-21T02:11:56.933ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1425, tv_nsec: 554990408 } })
72772025-03-21T02:11:56.933ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
72782025-03-21T02:11:56.934ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1425, tv_nsec: 556383822 } })
72792025-03-21T02:11:56.935ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
72802025-03-21T02:11:56.936ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72812025-03-21T02:11:56.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72822025-03-21T02:11:56.936ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
72832025-03-21T02:11:56.936ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
72842025-03-21T02:11:56.936ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
72852025-03-21T02:11:56.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
72862025-03-21T02:11:56.970ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
72872025-03-21T02:11:56.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
72882025-03-21T02:11:56.971ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
72892025-03-21T02:11:56.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
72902025-03-21T02:11:56.973ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72912025-03-21T02:11:56.974ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72922025-03-21T02:11:56.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72932025-03-21T02:11:56.975ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 54.831284ms iteration = 16 status = {"errors":[],"finish_invoked_ok":[]}
72942025-03-21T02:11:56.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
72952025-03-21T02:11:56.975ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72962025-03-21T02:11:56.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72972025-03-21T02:11:56.976ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72982025-03-21T02:11:56.977ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72992025-03-21T02:11:56.978ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
73002025-03-21T02:11:56.978ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
73012025-03-21T02:11:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
73022025-03-21T02:11:56.979ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 58.079331ms iteration = 16 status = {"errors":[],"garbage_collect_requested":[]}
73032025-03-21T02:11:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
73042025-03-21T02:11:56.979ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73052025-03-21T02:11:56.979ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
73062025-03-21T02:11:56.979ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73072025-03-21T02:11:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73082025-03-21T02:11:56.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
73092025-03-21T02:11:56.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73102025-03-21T02:11:56.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73112025-03-21T02:11:56.982ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
73122025-03-21T02:11:56.982ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
73132025-03-21T02:11:56.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
73142025-03-21T02:11:56.983ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
73152025-03-21T02:11:56.983ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
73162025-03-21T02:11:56.983ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 62.133292ms iteration = 16 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
73172025-03-21T02:11:56.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73182025-03-21T02:11:56.983ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 62.536259ms iteration = 16 status = {"errors":[],"requests_created_ok":[]}
73192025-03-21T02:11:56.984ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73202025-03-21T02:11:56.984ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73212025-03-21T02:11:56.984ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73222025-03-21T02:11:56.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73232025-03-21T02:11:56.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73242025-03-21T02:11:56.985ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.86435ms iteration = 46 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
73252025-03-21T02:11:56.985ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
73262025-03-21T02:11:56.986ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
73272025-03-21T02:11:56.986ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
73282025-03-21T02:11:56.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
73292025-03-21T02:11:56.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
73302025-03-21T02:11:56.987ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
73312025-03-21T02:11:56.987ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73322025-03-21T02:11:56.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
73332025-03-21T02:11:56.987ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73342025-03-21T02:11:56.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73352025-03-21T02:11:56.987ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73362025-03-21T02:11:56.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73372025-03-21T02:11:56.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73382025-03-21T02:11:56.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
73392025-03-21T02:11:56.988ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
73402025-03-21T02:11:56.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
73412025-03-21T02:11:56.989ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73422025-03-21T02:11:56.989ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
73432025-03-21T02:11:56.990ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 68.92729ms iteration = 16 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
73442025-03-21T02:11:56.990ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
73452025-03-21T02:11:56.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
73462025-03-21T02:11:56.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
73472025-03-21T02:11:56.991ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
73482025-03-21T02:11:56.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
73492025-03-21T02:11:56.991ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 70.63824ms iteration = 16 status = 0
73502025-03-21T02:11:56.991ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1425, tv_nsec: 613190651 } })
73512025-03-21T02:11:56.992ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
73522025-03-21T02:11:56.993ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
73532025-03-21T02:11:56.993ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 72.459122ms iteration = 16 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
73542025-03-21T02:11:57.001ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
73552025-03-21T02:11:57.001ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
73562025-03-21T02:11:57.001ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "de03b012-de39-4010-8e7a-c5c17a28f952", "content-length": "26", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
73572025-03-21T02:11:57.001ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
73582025-03-21T02:11:57.003ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
73592025-03-21T02:11:57.016ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
73602025-03-21T02:11:57.016ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 95.833488ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
73612025-03-21T02:11:57.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
73622025-03-21T02:11:57.019ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 98.308102ms iteration = 16 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
73632025-03-21T02:11:57.028ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73642025-03-21T02:11:57.030ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
73652025-03-21T02:11:57.030ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
73662025-03-21T02:11:57.030ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6138eb4b-aa79-43a1-bc1b-a8f2875c6d86", "content-length": "26", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
73672025-03-21T02:11:57.030ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
73682025-03-21T02:11:57.030ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
73692025-03-21T02:11:57.030ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73702025-03-21T02:11:57.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73712025-03-21T02:11:57.032ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
73722025-03-21T02:11:57.034ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 113.502386ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
73732025-03-21T02:11:57.039ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73742025-03-21T02:11:57.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73752025-03-21T02:11:57.041ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
73762025-03-21T02:11:57.041ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 120.851074ms iteration = 46 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
73772025-03-21T02:11:57.069ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73782025-03-21T02:11:57.070ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
73792025-03-21T02:11:57.070ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73802025-03-21T02:11:57.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73812025-03-21T02:11:57.072ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73822025-03-21T02:11:57.074ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
73832025-03-21T02:11:57.074ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73842025-03-21T02:11:57.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73852025-03-21T02:11:57.094ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 173.074293ms iteration = 17 status = {"physical_disks_added":0}
73862025-03-21T02:11:57.097ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
73872025-03-21T02:11:57.097ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
73882025-03-21T02:11:57.098ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0ac9438-0aca-455b-becc-b39b24fdf5ae", "content-length": "26", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
73892025-03-21T02:11:57.098ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
73902025-03-21T02:11:57.099ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
73912025-03-21T02:11:57.099ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
73922025-03-21T02:11:57.099ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6ae7c26-7fa1-4302-918a-31d8a12469a2", "content-length": "1", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
73932025-03-21T02:11:57.100ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
73942025-03-21T02:11:57.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
73952025-03-21T02:11:57.102ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 181.550251ms iteration = 16 status = null
73962025-03-21T02:11:57.143ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 222.87594ms iteration = 16 status = {}
73972025-03-21T02:11:57.182ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
73982025-03-21T02:11:57.182ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
73992025-03-21T02:11:57.182ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "249c9526-2ea9-4f5d-93a5-d6f971bd5f3f", "content-length": "372", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
74002025-03-21T02:11:57.182ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
74012025-03-21T02:11:57.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
74022025-03-21T02:11:57.184ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
74032025-03-21T02:11:57.184ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ccbb6b2-7f0d-4163-9336-62c88490bf45", "content-length": "691", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
74042025-03-21T02:11:57.184ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
74052025-03-21T02:11:57.184ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
74062025-03-21T02:11:57.184ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
74072025-03-21T02:11:57.186ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 265.214537ms iteration = 16 status = {}
74082025-03-21T02:11:57.228ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
74092025-03-21T02:11:57.228ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
74102025-03-21T02:11:57.229ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "4730a0aa-6b1a-4b14-a2c0-b771e065d57a", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
74112025-03-21T02:11:57.229ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
74122025-03-21T02:11:57.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
74132025-03-21T02:11:57.231ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
74142025-03-21T02:11:57.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
74152025-03-21T02:11:57.235ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
74162025-03-21T02:11:57.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
74172025-03-21T02:11:57.238ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
74182025-03-21T02:11:57.238ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
74192025-03-21T02:11:57.239ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "60c2ceaa-a763-4a9d-88dd-98ca123a8dcf", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
74202025-03-21T02:11:57.240ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 319.589934ms iteration = 16 status = {}
74212025-03-21T02:11:57.252ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74222025-03-21T02:11:57.253ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74232025-03-21T02:11:57.255ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
74242025-03-21T02:11:57.255ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74252025-03-21T02:11:57.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74262025-03-21T02:11:57.265ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
74272025-03-21T02:11:57.265ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "536216e3-9c69-4cdb-a27f-a54d8ea378ed", "content-length": "134", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
74282025-03-21T02:11:57.265ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
74292025-03-21T02:11:57.265ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
74302025-03-21T02:11:57.265ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
74312025-03-21T02:11:57.266ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
74322025-03-21T02:11:57.266ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74332025-03-21T02:11:57.266ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74342025-03-21T02:11:57.266ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74352025-03-21T02:11:57.266ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
74362025-03-21T02:11:57.266ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74372025-03-21T02:11:57.266ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
74382025-03-21T02:11:57.266ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
74392025-03-21T02:11:57.266ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74402025-03-21T02:11:57.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74412025-03-21T02:11:57.268ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
74422025-03-21T02:11:57.269ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc67660a-6ed5-4fe3-a7ee-c8f67aee6f6c", "content-length": "2", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
74432025-03-21T02:11:57.269ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
74442025-03-21T02:11:57.269ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b1cea4d-c507-4399-acf0-576ae2fc1f84", "content-length": "52", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
74452025-03-21T02:11:57.269ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
74462025-03-21T02:11:57.269ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
74472025-03-21T02:11:57.360ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "a64a6a78-4cc0-4262-b4dd-7030a9248328", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
74482025-03-21T02:11:57.360ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
74492025-03-21T02:11:57.360ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
74502025-03-21T02:11:57.360ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e317614-103c-4d53-981a-c6fe7e0ad6f4", "content-length": "682", "date": "Fri, 21 Mar 2025 02:11:56 GMT"} })
74512025-03-21T02:11:57.360ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74522025-03-21T02:11:57.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74532025-03-21T02:11:57.363ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74542025-03-21T02:11:57.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74552025-03-21T02:11:57.367ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74562025-03-21T02:11:57.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74572025-03-21T02:11:57.369ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74582025-03-21T02:11:57.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74592025-03-21T02:11:57.376ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 455.480419ms iteration = 16 status = {}
74602025-03-21T02:11:59.721ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 309 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = b07ccb77-22b2-47c0-a38b-4aff23312167 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
74612025-03-21T02:12:06.920ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 47 reason = Timeout
74622025-03-21T02:12:06.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74632025-03-21T02:12:06.920ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
74642025-03-21T02:12:06.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74652025-03-21T02:12:06.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74662025-03-21T02:12:06.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74672025-03-21T02:12:06.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74682025-03-21T02:12:06.924ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74692025-03-21T02:12:06.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74702025-03-21T02:12:06.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74712025-03-21T02:12:06.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74722025-03-21T02:12:06.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74732025-03-21T02:12:06.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74742025-03-21T02:12:06.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
74752025-03-21T02:12:06.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74762025-03-21T02:12:06.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74772025-03-21T02:12:06.939ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
74782025-03-21T02:12:06.939ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.855708ms iteration = 47 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
74792025-03-21T02:12:06.952ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74802025-03-21T02:12:06.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74812025-03-21T02:12:06.982ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.60003ms iteration = 47 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
74822025-03-21T02:12:09.721ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 433 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 2330db98-a5b3-4847-a9fb-4189209eecc1 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
74832025-03-21T02:12:16.921ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 48 reason = Timeout
74842025-03-21T02:12:16.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74852025-03-21T02:12:16.921ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
74862025-03-21T02:12:16.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74872025-03-21T02:12:16.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74882025-03-21T02:12:16.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74892025-03-21T02:12:16.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74902025-03-21T02:12:16.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74912025-03-21T02:12:16.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74922025-03-21T02:12:16.924ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.883648ms iteration = 48 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
74932025-03-21T02:12:16.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74942025-03-21T02:12:16.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74952025-03-21T02:12:16.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74962025-03-21T02:12:16.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74972025-03-21T02:12:16.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74982025-03-21T02:12:16.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74992025-03-21T02:12:16.931ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
75002025-03-21T02:12:16.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75012025-03-21T02:12:16.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75022025-03-21T02:12:16.940ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
75032025-03-21T02:12:16.940ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.398888ms iteration = 48 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
75042025-03-21T02:12:19.721ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 278 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = f65b6218-ed7b-452b-a8bf-ad7bb436fc90 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
75052025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 49 reason = Timeout
75062025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
75072025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
75082025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 17 reason = Timeout
75092025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
75102025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
75112025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
75122025-03-21T02:12:26.920ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
75132025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 18 reason = Timeout
75142025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 24.43\u{b5}s iteration = 10 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
75152025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75162025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 17 reason = Timeout
75172025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
75182025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
75192025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
75202025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
75212025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
75222025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 17 reason = Timeout
75232025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
75242025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
75252025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75262025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
75272025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
75282025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
75292025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 9 reason = Timeout
75302025-03-21T02:12:26.920ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
75312025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 17 reason = Timeout
75322025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
75332025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
75342025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
75352025-03-21T02:12:26.920ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 9 reason = Timeout
75362025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
75372025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
75382025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
75392025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 17 reason = Timeout
75402025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
75412025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
75422025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 9 reason = Timeout
75432025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
75442025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
75452025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
75462025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
75472025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
75482025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
75492025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
75502025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
75512025-03-21T02:12:26.921ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 02:02:26.921050429 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
75522025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 17 reason = Timeout
75532025-03-21T02:12:26.921ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
75542025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
75552025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 17 reason = Timeout
75562025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75572025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
75582025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
75592025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
75602025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 9 reason = Timeout
75612025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
75622025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 17 reason = Timeout
75632025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
75642025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 17 reason = Timeout
75652025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
75662025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
75672025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75682025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 17 reason = Timeout
75692025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
75702025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75712025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75722025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
75732025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
75742025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
75752025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
75762025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
75772025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
75782025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
75792025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
75802025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75812025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
75822025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
75832025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
75842025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
75852025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
75862025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
75872025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
75882025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75892025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
75902025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
75912025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
75922025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
75932025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75942025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
75952025-03-21T02:12:26.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75962025-03-21T02:12:26.922ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1455, tv_nsec: 543033320 } })
75972025-03-21T02:12:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
75982025-03-21T02:12:26.922ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1455, tv_nsec: 543094651 } })
75992025-03-21T02:12:26.922ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
76002025-03-21T02:12:26.922ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.463235ms iteration = 9 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
76012025-03-21T02:12:26.922ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
76022025-03-21T02:12:26.922ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.569158ms iteration = 9 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
76032025-03-21T02:12:26.923ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.662426ms iteration = 17 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
76042025-03-21T02:12:26.925ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76052025-03-21T02:12:26.926ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76062025-03-21T02:12:26.926ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
76072025-03-21T02:12:26.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
76082025-03-21T02:12:26.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
76092025-03-21T02:12:26.927ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
76102025-03-21T02:12:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
76112025-03-21T02:12:26.928ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1455, tv_nsec: 549746757 } })
76122025-03-21T02:12:26.928ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
76132025-03-21T02:12:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
76142025-03-21T02:12:26.929ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76152025-03-21T02:12:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76162025-03-21T02:12:26.931ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1455, tv_nsec: 552489994 } })
76172025-03-21T02:12:26.931ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
76182025-03-21T02:12:26.933ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
76192025-03-21T02:12:26.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
76202025-03-21T02:12:26.934ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76212025-03-21T02:12:26.935ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 14.456291ms iteration = 9 status = {"errors":[],"found_port_ok":[]}
76222025-03-21T02:12:26.935ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
76232025-03-21T02:12:26.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
76242025-03-21T02:12:26.937ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
76252025-03-21T02:12:26.937ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
76262025-03-21T02:12:26.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
76272025-03-21T02:12:26.938ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
76282025-03-21T02:12:26.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
76292025-03-21T02:12:26.940ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76302025-03-21T02:12:26.941ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 02:02:26.921050429 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
76312025-03-21T02:12:26.941ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 20.762002ms iteration = 9 status = {"expiration":"2025-03-21T02:02:26.921050429Z","pruned":[]}
76322025-03-21T02:12:26.943ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
76332025-03-21T02:12:26.943ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76342025-03-21T02:12:26.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76352025-03-21T02:12:26.946ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
76362025-03-21T02:12:26.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
76372025-03-21T02:12:26.947ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76382025-03-21T02:12:26.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
76392025-03-21T02:12:26.950ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76402025-03-21T02:12:26.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76412025-03-21T02:12:26.954ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 33.487043ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
76422025-03-21T02:12:26.955ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
76432025-03-21T02:12:26.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
76442025-03-21T02:12:26.960ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
76452025-03-21T02:12:26.960ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 39.556929ms iteration = 17 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
76462025-03-21T02:12:26.962ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76472025-03-21T02:12:26.963ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76482025-03-21T02:12:26.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76492025-03-21T02:12:26.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
76502025-03-21T02:12:26.976ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76512025-03-21T02:12:26.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76522025-03-21T02:12:26.980ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76532025-03-21T02:12:26.984ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
76542025-03-21T02:12:26.984ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
76552025-03-21T02:12:26.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
76562025-03-21T02:12:26.988ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76572025-03-21T02:12:26.991ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 70.190832ms iteration = 17 status = {"errors":[],"finish_invoked_ok":[]}
76582025-03-21T02:12:26.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
76592025-03-21T02:12:26.991ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76602025-03-21T02:12:26.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76612025-03-21T02:12:26.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76622025-03-21T02:12:26.998ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 77.5874ms iteration = 17 status = {"errors":[],"requests_created_ok":[]}
76632025-03-21T02:12:26.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
76642025-03-21T02:12:26.999ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76652025-03-21T02:12:26.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76662025-03-21T02:12:26.999ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76672025-03-21T02:12:27.000ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76682025-03-21T02:12:27.000ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
76692025-03-21T02:12:27.000ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
76702025-03-21T02:12:27.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
76712025-03-21T02:12:27.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
76722025-03-21T02:12:27.002ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76732025-03-21T02:12:27.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76742025-03-21T02:12:27.002ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
76752025-03-21T02:12:27.002ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
76762025-03-21T02:12:27.002ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 81.388677ms iteration = 9 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
76772025-03-21T02:12:27.005ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76782025-03-21T02:12:27.006ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76792025-03-21T02:12:27.006ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
76802025-03-21T02:12:27.006ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
76812025-03-21T02:12:27.006ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1455, tv_nsec: 627879147 } })
76822025-03-21T02:12:27.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
76832025-03-21T02:12:27.007ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
76842025-03-21T02:12:27.008ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
76852025-03-21T02:12:27.008ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
76862025-03-21T02:12:27.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
76872025-03-21T02:12:27.012ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76882025-03-21T02:12:27.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76892025-03-21T02:12:27.014ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
76902025-03-21T02:12:27.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
76912025-03-21T02:12:27.016ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 96.195675ms iteration = 49 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
76922025-03-21T02:12:27.017ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76932025-03-21T02:12:27.019ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76942025-03-21T02:12:27.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
76952025-03-21T02:12:27.021ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76962025-03-21T02:12:27.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76972025-03-21T02:12:27.023ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
76982025-03-21T02:12:27.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
76992025-03-21T02:12:27.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
77002025-03-21T02:12:27.027ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
77012025-03-21T02:12:27.027ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
77022025-03-21T02:12:27.027ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
77032025-03-21T02:12:27.027ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
77042025-03-21T02:12:27.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
77052025-03-21T02:12:27.029ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
77062025-03-21T02:12:27.029ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
77072025-03-21T02:12:27.029ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
77082025-03-21T02:12:27.029ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
77092025-03-21T02:12:27.029ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 108.665342ms iteration = 17 status = {"errors":[],"garbage_collect_requested":[]}
77102025-03-21T02:12:27.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
77112025-03-21T02:12:27.029ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77122025-03-21T02:12:27.031ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
77132025-03-21T02:12:27.031ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77142025-03-21T02:12:27.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
77152025-03-21T02:12:27.034ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
77162025-03-21T02:12:27.034ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
77172025-03-21T02:12:27.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
77182025-03-21T02:12:27.034ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
77192025-03-21T02:12:27.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
77202025-03-21T02:12:27.035ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
77212025-03-21T02:12:27.035ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
77222025-03-21T02:12:27.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
77232025-03-21T02:12:27.038ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77242025-03-21T02:12:27.039ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
77252025-03-21T02:12:27.039ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 118.31862ms iteration = 49 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
77262025-03-21T02:12:27.039ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
77272025-03-21T02:12:27.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
77282025-03-21T02:12:27.040ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
77292025-03-21T02:12:27.040ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
77302025-03-21T02:12:27.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
77312025-03-21T02:12:27.042ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
77322025-03-21T02:12:27.042ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
77332025-03-21T02:12:27.042ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
77342025-03-21T02:12:27.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
77352025-03-21T02:12:27.042ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77362025-03-21T02:12:27.043ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77372025-03-21T02:12:27.045ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
77382025-03-21T02:12:27.045ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77392025-03-21T02:12:27.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77402025-03-21T02:12:27.046ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
77412025-03-21T02:12:27.046ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
77422025-03-21T02:12:27.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
77432025-03-21T02:12:27.047ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77442025-03-21T02:12:27.051ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
77452025-03-21T02:12:27.051ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
77462025-03-21T02:12:27.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
77472025-03-21T02:12:27.057ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
77482025-03-21T02:12:27.057ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
77492025-03-21T02:12:27.057ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-21 02:04:30.886360 UTC dns_group = internal generation = 1
77502025-03-21T02:12:27.057ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 136.773961ms iteration = 9 status = {"generation":1}
77512025-03-21T02:12:27.058ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
77522025-03-21T02:12:27.058ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
77532025-03-21T02:12:27.058ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 137.422762ms iteration = 17 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
77542025-03-21T02:12:27.076ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 155.438836ms iteration = 18 status = {"physical_disks_added":0}
77552025-03-21T02:12:27.077ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
77562025-03-21T02:12:27.077ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
77572025-03-21T02:12:27.077ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 156.619907ms iteration = 9 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
77582025-03-21T02:12:27.078ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
77592025-03-21T02:12:27.078ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
77602025-03-21T02:12:27.078ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
77612025-03-21T02:12:27.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
77622025-03-21T02:12:27.080ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
77632025-03-21T02:12:27.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
77642025-03-21T02:12:27.083ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 162.298105ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
77652025-03-21T02:12:27.083ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77662025-03-21T02:12:27.084ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 163.387605ms iteration = 17 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
77672025-03-21T02:12:27.087ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
77682025-03-21T02:12:27.087ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
77692025-03-21T02:12:27.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
77702025-03-21T02:12:27.090ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77712025-03-21T02:12:27.092ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
77722025-03-21T02:12:27.092ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
77732025-03-21T02:12:27.092ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-21 02:04:31.662541 UTC dns_group = external generation = 2
77742025-03-21T02:12:27.092ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 171.242011ms iteration = 9 status = {"generation":2}
77752025-03-21T02:12:27.093ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
77762025-03-21T02:12:27.093ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
77772025-03-21T02:12:27.093ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
77782025-03-21T02:12:27.095ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 174.07102ms iteration = 17 status = 0
77792025-03-21T02:12:27.098ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
77802025-03-21T02:12:27.099ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
77812025-03-21T02:12:27.099ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
77822025-03-21T02:12:27.099ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
77832025-03-21T02:12:27.099ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
77842025-03-21T02:12:27.099ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "39e4c5f9-7549-4d99-816c-a81c7730eda6", "content-length": "26", "date": "Fri, 21 Mar 2025 02:12:26 GMT"} })
77852025-03-21T02:12:27.099ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
77862025-03-21T02:12:27.101ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
77872025-03-21T02:12:27.103ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) }
77882025-03-21T02:12:27.103ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 183.016366ms iteration = 9 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
77892025-03-21T02:12:27.104ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77902025-03-21T02:12:27.106ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
77912025-03-21T02:12:27.106ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
77922025-03-21T02:12:27.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
77932025-03-21T02:12:27.109ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
77942025-03-21T02:12:27.109ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 188.071544ms iteration = 17 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
77952025-03-21T02:12:27.131ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
77962025-03-21T02:12:27.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
77972025-03-21T02:12:27.163ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
77982025-03-21T02:12:27.163ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
77992025-03-21T02:12:27.164ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "900db1f4-e707-47b2-8e87-875306b69cfa", "content-length": "26", "date": "Fri, 21 Mar 2025 02:12:26 GMT"} })
78002025-03-21T02:12:27.164ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
78012025-03-21T02:12:27.166ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
78022025-03-21T02:12:27.185ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
78032025-03-21T02:12:27.185ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 263.928095ms iteration = 9 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
78042025-03-21T02:12:27.227ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
78052025-03-21T02:12:27.234ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
78062025-03-21T02:12:27.234ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
78072025-03-21T02:12:27.234ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "306adad4-b7e1-427d-9439-7dd107783133", "content-length": "26", "date": "Fri, 21 Mar 2025 02:12:26 GMT"} })
78082025-03-21T02:12:27.234ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
78092025-03-21T02:12:27.235ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
78102025-03-21T02:12:27.248ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78112025-03-21T02:12:27.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78122025-03-21T02:12:27.251ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78132025-03-21T02:12:27.252ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
78142025-03-21T02:12:27.252ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78152025-03-21T02:12:27.252ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78162025-03-21T02:12:27.295ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 374.702443ms iteration = 17 status = {}
78172025-03-21T02:12:27.299ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
78182025-03-21T02:12:27.299ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
78192025-03-21T02:12:27.300ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "0240677a-24b9-43b4-b8e6-471f744e0be2", "content-length": "372", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} })
78202025-03-21T02:12:27.300ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
78212025-03-21T02:12:27.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
78222025-03-21T02:12:27.302ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78232025-03-21T02:12:27.303ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
78242025-03-21T02:12:27.303ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78252025-03-21T02:12:27.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78262025-03-21T02:12:27.315ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
78272025-03-21T02:12:27.315ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
78282025-03-21T02:12:27.316ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "71bcd442-b775-4e0f-bb46-52c2cde28c3e", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} })
78292025-03-21T02:12:27.316ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
78302025-03-21T02:12:27.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
78312025-03-21T02:12:27.318ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
78322025-03-21T02:12:27.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
78332025-03-21T02:12:27.321ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
78342025-03-21T02:12:27.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
78352025-03-21T02:12:27.324ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
78362025-03-21T02:12:27.324ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
78372025-03-21T02:12:27.330ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
78382025-03-21T02:12:27.330ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "9c4245ee-3ed5-4267-a716-958904f3b695", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
78392025-03-21T02:12:27.330ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
78402025-03-21T02:12:27.331ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "a6cf6495-da22-4b7f-ace3-4a26b1f95b5e", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
78412025-03-21T02:12:27.331ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
78422025-03-21T02:12:27.333ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
78432025-03-21T02:12:27.333ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 412.199776ms iteration = 11 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
78442025-03-21T02:12:27.402ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "46fd1ea1-9e3c-45ae-9d24-1a868530d456", "date": "Fri, 21 Mar 2025 02:12:26 GMT"} })
78452025-03-21T02:12:27.404ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 483.519848ms iteration = 17 status = {}
78462025-03-21T02:12:27.416ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
78472025-03-21T02:12:27.416ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ebb1580-5721-4b05-86d5-ec16e3a8b98e", "content-length": "1", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} })
78482025-03-21T02:12:27.418ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
78492025-03-21T02:12:27.419ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
78502025-03-21T02:12:27.420ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 499.683399ms iteration = 17 status = null
78512025-03-21T02:12:27.537ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
78522025-03-21T02:12:27.619ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
78532025-03-21T02:12:27.619ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "de6418a4-cb3f-4620-bb73-9bdf726326cc", "content-length": "691", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} })
78542025-03-21T02:12:27.619ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
78552025-03-21T02:12:27.619ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
78562025-03-21T02:12:27.619ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
78572025-03-21T02:12:27.621ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 700.661877ms iteration = 17 status = {}
78582025-03-21T02:12:27.703ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
78592025-03-21T02:12:27.704ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "8e57c4fe-a033-429f-8d9c-e537cde112ad", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
78602025-03-21T02:12:27.704ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
78612025-03-21T02:12:27.705ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "e8ea0691-72ee-4b27-a8b4-7436939bb350", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
78622025-03-21T02:12:27.705ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
78632025-03-21T02:12:27.707ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "6808859c-a3da-447f-a305-119c81d9fde4", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
78642025-03-21T02:12:27.707ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
78652025-03-21T02:12:27.709ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
78662025-03-21T02:12:27.709ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 788.70185ms iteration = 11 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
78672025-03-21T02:12:27.732ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78682025-03-21T02:12:27.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78692025-03-21T02:12:27.734ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
78702025-03-21T02:12:27.734ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78712025-03-21T02:12:27.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78722025-03-21T02:12:27.745ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
78732025-03-21T02:12:27.745ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "83d22e7f-c401-4502-a5ef-3fd829bbff22", "content-length": "134", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} })
78742025-03-21T02:12:27.745ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
78752025-03-21T02:12:27.745ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
78762025-03-21T02:12:27.745ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
78772025-03-21T02:12:27.745ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
78782025-03-21T02:12:27.745ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
78792025-03-21T02:12:27.745ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
78802025-03-21T02:12:27.745ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
78812025-03-21T02:12:27.745ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
78822025-03-21T02:12:27.745ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
78832025-03-21T02:12:27.745ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
78842025-03-21T02:12:27.746ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
78852025-03-21T02:12:27.746ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78862025-03-21T02:12:27.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78872025-03-21T02:12:27.748ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
78882025-03-21T02:12:27.748ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8eb60a7-6764-408b-a0bd-effab5e4733d", "content-length": "2", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} })
78892025-03-21T02:12:27.748ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
78902025-03-21T02:12:27.748ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4a6d7a3-984d-4c35-a494-36d240cfb956", "content-length": "52", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} })
78912025-03-21T02:12:27.748ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
78922025-03-21T02:12:27.748ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
78932025-03-21T02:12:27.841ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "328926c8-a1e9-4efc-9db1-49b126f88e92", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} })
78942025-03-21T02:12:27.841ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
78952025-03-21T02:12:27.841ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
78962025-03-21T02:12:27.841ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "03ba9fe6-8271-4cda-93aa-18a22504174c", "content-length": "682", "date": "Fri, 21 Mar 2025 02:12:27 GMT"} })
78972025-03-21T02:12:27.841ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78982025-03-21T02:12:27.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78992025-03-21T02:12:27.845ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
79002025-03-21T02:12:27.845ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
79012025-03-21T02:12:27.849ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
79022025-03-21T02:12:27.849ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
79032025-03-21T02:12:27.851ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
79042025-03-21T02:12:27.851ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
79052025-03-21T02:12:27.859ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 938.400416ms iteration = 17 status = {}
79062025-03-21T02:12:29.721ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 285 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = ee082c49-2005-4c7d-8566-591bf92311f6 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
79072025-03-21T02:12:36.921ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 50 reason = Timeout
79082025-03-21T02:12:36.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
79092025-03-21T02:12:36.921ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
79102025-03-21T02:12:36.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79112025-03-21T02:12:36.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79122025-03-21T02:12:36.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
79132025-03-21T02:12:36.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79142025-03-21T02:12:36.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
79152025-03-21T02:12:36.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
79162025-03-21T02:12:36.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
79172025-03-21T02:12:36.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79182025-03-21T02:12:36.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79192025-03-21T02:12:36.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79202025-03-21T02:12:36.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79212025-03-21T02:12:36.953ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.683899ms iteration = 50 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
79222025-03-21T02:12:36.956ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79232025-03-21T02:12:36.959ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
79242025-03-21T02:12:36.959ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79252025-03-21T02:12:36.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79262025-03-21T02:12:36.966ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
79272025-03-21T02:12:36.966ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 45.069854ms iteration = 50 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
79282025-03-21T02:12:39.722ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 296 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = df236ae2-a7e1-4b8f-9f18-fa3adf078664 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
79292025-03-21T02:12:46.921ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 51 reason = Timeout
79302025-03-21T02:12:46.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
79312025-03-21T02:12:46.921ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
79322025-03-21T02:12:46.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79332025-03-21T02:12:46.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
79342025-03-21T02:12:46.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79352025-03-21T02:12:46.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
79362025-03-21T02:12:46.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
79372025-03-21T02:12:46.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79382025-03-21T02:12:46.925ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.432387ms iteration = 51 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
79392025-03-21T02:12:46.926ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
79402025-03-21T02:12:46.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79412025-03-21T02:12:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79422025-03-21T02:12:46.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79432025-03-21T02:12:46.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79442025-03-21T02:12:46.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79452025-03-21T02:12:46.932ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
79462025-03-21T02:12:46.933ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79472025-03-21T02:12:46.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79482025-03-21T02:12:46.942ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
79492025-03-21T02:12:46.942ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.28529ms iteration = 51 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
79502025-03-21T02:12:49.722ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 290 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 6d940c62-bcfd-4602-bb8e-b5b4da6e9c1d response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
79512025-03-21T02:12:54.712ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:56306
79522025-03-21T02:12:54.712ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56306 req_id = f35f5952-89e6-4158-a455-2a78ca8e8eb8 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
79532025-03-21T02:12:54.713ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56306 req_id = f35f5952-89e6-4158-a455-2a78ca8e8eb8 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
79542025-03-21T02:12:54.715ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3248 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56306 req_id = f35f5952-89e6-4158-a455-2a78ca8e8eb8 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
79552025-03-21T02:12:54.716ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56306 req_id = 5b7cab22-cb10-40c6-92ea-cf1f4e8aacbe roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
79562025-03-21T02:12:54.716ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56306 req_id = 5b7cab22-cb10-40c6-92ea-cf1f4e8aacbe resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
79572025-03-21T02:12:54.718ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2546 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56306 req_id = 5b7cab22-cb10-40c6-92ea-cf1f4e8aacbe response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
79582025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 52 reason = Timeout
79592025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
79602025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
79612025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
79622025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
79632025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79642025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 18 reason = Timeout
79652025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 18 reason = Timeout
79662025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
79672025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
79682025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79692025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 18 reason = Timeout
79702025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
79712025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 18 reason = Timeout
79722025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
79732025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 18 reason = Timeout
79742025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
79752025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
79762025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
79772025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
79782025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 19 reason = Timeout
79792025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
79802025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
79812025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
79822025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 18 reason = Timeout
79832025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 18 reason = Timeout
79842025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
79852025-03-21T02:12:56.921ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
79862025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
79872025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 18 reason = Timeout
79882025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
79892025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
79902025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 18 reason = Timeout
79912025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
79922025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
79932025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 18 reason = Timeout
79942025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
79952025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
79962025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
79972025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
79982025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
79992025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
80002025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80012025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
80022025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80032025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
80042025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
80052025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
80062025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80072025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80082025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
80092025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
80102025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
80112025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
80122025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
80132025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
80142025-03-21T02:12:56.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
80152025-03-21T02:12:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
80162025-03-21T02:12:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
80172025-03-21T02:12:56.924ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80182025-03-21T02:12:56.924ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.891198ms iteration = 18 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
80192025-03-21T02:12:56.925ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80202025-03-21T02:12:56.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
80212025-03-21T02:12:56.927ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
80222025-03-21T02:12:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
80232025-03-21T02:12:56.928ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80242025-03-21T02:12:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
80252025-03-21T02:12:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80262025-03-21T02:12:56.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80272025-03-21T02:12:56.931ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80282025-03-21T02:12:56.932ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80292025-03-21T02:12:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
80302025-03-21T02:12:56.934ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
80312025-03-21T02:12:56.934ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 13.492195ms iteration = 18 status = {"errors":[],"requests_created_ok":[]}
80322025-03-21T02:12:56.934ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
80332025-03-21T02:12:56.934ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80342025-03-21T02:12:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80352025-03-21T02:12:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80362025-03-21T02:12:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
80372025-03-21T02:12:56.934ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
80382025-03-21T02:12:56.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
80392025-03-21T02:12:56.936ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80402025-03-21T02:12:56.936ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 15.842196ms iteration = 18 status = {"errors":[],"finish_invoked_ok":[]}
80412025-03-21T02:12:56.937ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
80422025-03-21T02:12:56.937ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
80432025-03-21T02:12:56.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
80442025-03-21T02:12:56.941ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1485, tv_nsec: 562261838 } })
80452025-03-21T02:12:56.941ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
80462025-03-21T02:12:56.942ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1485, tv_nsec: 563795045 } })
80472025-03-21T02:12:56.943ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
80482025-03-21T02:12:56.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
80492025-03-21T02:12:56.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80502025-03-21T02:12:56.947ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1485, tv_nsec: 568658839 } })
80512025-03-21T02:12:56.949ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
80522025-03-21T02:12:56.951ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80532025-03-21T02:12:56.952ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
80542025-03-21T02:12:56.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
80552025-03-21T02:12:56.953ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
80562025-03-21T02:12:56.953ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80572025-03-21T02:12:56.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80582025-03-21T02:12:56.954ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80592025-03-21T02:12:56.957ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
80602025-03-21T02:12:56.957ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
80612025-03-21T02:12:56.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
80622025-03-21T02:12:56.979ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
80632025-03-21T02:12:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
80642025-03-21T02:12:56.981ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
80652025-03-21T02:12:56.981ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
80662025-03-21T02:12:56.981ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
80672025-03-21T02:12:56.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
80682025-03-21T02:12:56.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
80692025-03-21T02:12:56.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
80702025-03-21T02:12:56.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 65.535651ms iteration = 18 status = {"errors":[],"garbage_collect_requested":[]}
80712025-03-21T02:12:56.991ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80722025-03-21T02:12:56.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80732025-03-21T02:12:56.992ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
80742025-03-21T02:12:56.993ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 72.104366ms iteration = 52 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
80752025-03-21T02:12:56.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
80762025-03-21T02:12:56.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
80772025-03-21T02:12:56.994ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
80782025-03-21T02:12:56.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80792025-03-21T02:12:56.995ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
80802025-03-21T02:12:56.995ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 73.896306ms iteration = 18 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
80812025-03-21T02:12:56.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
80822025-03-21T02:12:56.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
80832025-03-21T02:12:56.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80842025-03-21T02:12:56.995ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
80852025-03-21T02:12:56.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
80862025-03-21T02:12:56.996ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 75.364272ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
80872025-03-21T02:12:56.997ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
80882025-03-21T02:12:56.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80892025-03-21T02:12:57.003ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
80902025-03-21T02:12:57.003ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 82.261192ms iteration = 52 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
80912025-03-21T02:12:57.015ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
80922025-03-21T02:12:57.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
80932025-03-21T02:12:57.016ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
80942025-03-21T02:12:57.016ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
80952025-03-21T02:12:57.016ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
80962025-03-21T02:12:57.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
80972025-03-21T02:12:57.019ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
80982025-03-21T02:12:57.019ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
80992025-03-21T02:12:57.019ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e927a06-2e79-4e44-bbe0-68664b963414", "content-length": "26", "date": "Fri, 21 Mar 2025 02:12:56 GMT"} })
81002025-03-21T02:12:57.019ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
81012025-03-21T02:12:57.019ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 98.659488ms iteration = 19 status = {"physical_disks_added":0}
81022025-03-21T02:12:57.019ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 98.539115ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
81032025-03-21T02:12:57.021ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
81042025-03-21T02:12:57.025ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
81052025-03-21T02:12:57.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
81062025-03-21T02:12:57.028ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
81072025-03-21T02:12:57.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
81082025-03-21T02:12:57.030ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81092025-03-21T02:12:57.031ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
81102025-03-21T02:12:57.031ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
81112025-03-21T02:12:57.031ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "892e912a-f3d7-4de8-ba73-b7f23215e847", "content-length": "26", "date": "Fri, 21 Mar 2025 02:12:57 GMT"} })
81122025-03-21T02:12:57.031ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
81132025-03-21T02:12:57.033ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
81142025-03-21T02:12:57.033ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
81152025-03-21T02:12:57.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
81162025-03-21T02:12:57.033ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
81172025-03-21T02:12:57.034ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81182025-03-21T02:12:57.034ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 113.603778ms iteration = 18 status = 0
81192025-03-21T02:12:57.035ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
81202025-03-21T02:12:57.035ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
81212025-03-21T02:12:57.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
81222025-03-21T02:12:57.048ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
81232025-03-21T02:12:57.048ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
81242025-03-21T02:12:57.049ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa70c151-038e-4482-b62f-266706eb445e", "content-length": "26", "date": "Fri, 21 Mar 2025 02:12:57 GMT"} })
81252025-03-21T02:12:57.049ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
81262025-03-21T02:12:57.050ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 129.717058ms iteration = 18 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
81272025-03-21T02:12:57.051ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
81282025-03-21T02:12:57.052ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
81292025-03-21T02:12:57.052ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
81302025-03-21T02:12:57.052ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 131.595591ms iteration = 18 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
81312025-03-21T02:12:57.053ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
81322025-03-21T02:12:57.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
81332025-03-21T02:12:57.135ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
81342025-03-21T02:12:57.136ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8bb4c39-4f82-4b0b-958c-0f19cae2cb90", "content-length": "1", "date": "Fri, 21 Mar 2025 02:12:56 GMT"} })
81352025-03-21T02:12:57.137ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
81362025-03-21T02:12:57.137ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
81372025-03-21T02:12:57.153ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
81382025-03-21T02:12:57.153ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
81392025-03-21T02:12:57.153ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "f87a377a-d482-42c1-a2a2-a4832755546d", "content-length": "372", "date": "Fri, 21 Mar 2025 02:12:57 GMT"} })
81402025-03-21T02:12:57.153ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81412025-03-21T02:12:57.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81422025-03-21T02:12:57.156ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 235.648472ms iteration = 18 status = null
81432025-03-21T02:12:57.163ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 242.750986ms iteration = 18 status = {}
81442025-03-21T02:12:57.186ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
81452025-03-21T02:12:57.186ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 265.620544ms iteration = 18 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
81462025-03-21T02:12:57.201ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
81472025-03-21T02:12:57.201ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
81482025-03-21T02:12:57.202ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "0573ed15-0185-4d24-ad3d-0adf526cafd9", "date": "Fri, 21 Mar 2025 02:12:57 GMT"} })
81492025-03-21T02:12:57.202ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81502025-03-21T02:12:57.202ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81512025-03-21T02:12:57.204ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81522025-03-21T02:12:57.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81532025-03-21T02:12:57.207ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81542025-03-21T02:12:57.207ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81552025-03-21T02:12:57.211ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
81562025-03-21T02:12:57.211ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
81572025-03-21T02:12:57.212ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "37901ac2-3c85-45f9-9846-de6014eb5f0f", "date": "Fri, 21 Mar 2025 02:12:57 GMT"} })
81582025-03-21T02:12:57.213ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 292.680496ms iteration = 18 status = {}
81592025-03-21T02:12:57.221ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81602025-03-21T02:12:57.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81612025-03-21T02:12:57.223ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
81622025-03-21T02:12:57.223ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81632025-03-21T02:12:57.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81642025-03-21T02:12:57.234ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
81652025-03-21T02:12:57.234ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "cce1bbed-edda-45ba-b5e6-a58003e0f3f8", "content-length": "134", "date": "Fri, 21 Mar 2025 02:12:56 GMT"} })
81662025-03-21T02:12:57.234ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
81672025-03-21T02:12:57.234ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
81682025-03-21T02:12:57.234ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
81692025-03-21T02:12:57.234ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
81702025-03-21T02:12:57.234ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81712025-03-21T02:12:57.234ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81722025-03-21T02:12:57.234ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81732025-03-21T02:12:57.234ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
81742025-03-21T02:12:57.234ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81752025-03-21T02:12:57.234ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
81762025-03-21T02:12:57.235ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
81772025-03-21T02:12:57.235ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81782025-03-21T02:12:57.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81792025-03-21T02:12:57.237ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
81802025-03-21T02:12:57.237ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "4491b00c-8425-4818-bad7-14c151cc7930", "content-length": "2", "date": "Fri, 21 Mar 2025 02:12:56 GMT"} })
81812025-03-21T02:12:57.237ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
81822025-03-21T02:12:57.237ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "176adaa6-01cb-484e-8352-7102750aee68", "content-length": "52", "date": "Fri, 21 Mar 2025 02:12:56 GMT"} })
81832025-03-21T02:12:57.237ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
81842025-03-21T02:12:57.237ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
81852025-03-21T02:12:57.258ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
81862025-03-21T02:12:57.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
81872025-03-21T02:12:57.290ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81882025-03-21T02:12:57.291ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
81892025-03-21T02:12:57.291ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
81902025-03-21T02:12:57.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
81912025-03-21T02:12:57.293ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81922025-03-21T02:12:57.295ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
81932025-03-21T02:12:57.295ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
81942025-03-21T02:12:57.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
81952025-03-21T02:12:57.344ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "3473a616-f609-41be-b437-6c50c3c1e1e0", "date": "Fri, 21 Mar 2025 02:12:57 GMT"} })
81962025-03-21T02:12:57.344ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
81972025-03-21T02:12:57.344ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
81982025-03-21T02:12:57.344ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "44a0eeaf-317e-4e3f-84a5-efa8de4c9d43", "content-length": "682", "date": "Fri, 21 Mar 2025 02:12:57 GMT"} })
81992025-03-21T02:12:57.344ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82002025-03-21T02:12:57.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82012025-03-21T02:12:57.347ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82022025-03-21T02:12:57.347ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82032025-03-21T02:12:57.351ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82042025-03-21T02:12:57.351ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82052025-03-21T02:12:57.353ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82062025-03-21T02:12:57.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82072025-03-21T02:12:57.361ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 439.755155ms iteration = 18 status = {}
82082025-03-21T02:12:57.390ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
82092025-03-21T02:12:57.391ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "87263dfe-6083-4fe4-b9ef-4a3580562d1c", "content-length": "691", "date": "Fri, 21 Mar 2025 02:12:57 GMT"} })
82102025-03-21T02:12:57.391ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
82112025-03-21T02:12:57.391ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
82122025-03-21T02:12:57.391ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
82132025-03-21T02:12:57.393ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 471.900625ms iteration = 18 status = {}
82142025-03-21T02:12:57.792ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
82152025-03-21T02:12:57.792ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
82162025-03-21T02:12:57.822ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
82172025-03-21T02:12:57.822ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:55163
82182025-03-21T02:12:57.823ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:55163 req_id = 1f86e800-daa4-4a94-8451-d43037972567 roles = RoleSet { roles: {} } uri = /metrics/producers
82192025-03-21T02:12:57.823ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:55163 req_id = 1f86e800-daa4-4a94-8451-d43037972567 resource = Database result = Ok(()) uri = /metrics/producers
82202025-03-21T02:12:57.865ZINFOnexus: assigned collector to new producer collector_id = a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53 file = nexus/src/app/oximeter.rs:76 producer_id = e05eb799-8cf0-4702-bd3b-9994ae86118a
82212025-03-21T02:12:57.865ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 42878 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:55163 req_id = 1f86e800-daa4-4a94-8451-d43037972567 response_code = 201 uri = /metrics/producers
82222025-03-21T02:12:57.865ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "1f86e800-daa4-4a94-8451-d43037972567", "content-length": "41", "date": "Fri, 21 Mar 2025 02:12:57 GMT"} })
82232025-03-21T02:12:57.867ZDEBGnexus (producer-registration-task): registered with nexus successfully
82242025-03-21T02:12:57.867ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
82252025-03-21T02:12:59.722ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 294 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = b7295eb3-9799-41ba-a215-9db843ed53de response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
82262025-03-21T02:13:06.921ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 53 reason = Timeout
82272025-03-21T02:13:06.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82282025-03-21T02:13:06.921ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
82292025-03-21T02:13:06.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82302025-03-21T02:13:06.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82312025-03-21T02:13:06.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82322025-03-21T02:13:06.950ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82332025-03-21T02:13:06.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82342025-03-21T02:13:06.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82352025-03-21T02:13:06.954ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
82362025-03-21T02:13:06.954ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82372025-03-21T02:13:06.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82382025-03-21T02:13:06.957ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82392025-03-21T02:13:06.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82402025-03-21T02:13:06.958ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82412025-03-21T02:13:06.961ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
82422025-03-21T02:13:06.961ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82432025-03-21T02:13:06.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82442025-03-21T02:13:06.970ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
82452025-03-21T02:13:06.970ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 49.224826ms iteration = 53 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
82462025-03-21T02:13:06.980ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 58.689802ms iteration = 53 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
82472025-03-21T02:13:09.721ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = cfb42f44-0f67-44d2-b0e2-3944dac20d5c response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
82482025-03-21T02:13:16.921ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 54 reason = Timeout
82492025-03-21T02:13:16.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82502025-03-21T02:13:16.921ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
82512025-03-21T02:13:16.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82522025-03-21T02:13:16.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82532025-03-21T02:13:16.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82542025-03-21T02:13:16.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82552025-03-21T02:13:16.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82562025-03-21T02:13:16.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82572025-03-21T02:13:16.926ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.811814ms iteration = 54 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
82582025-03-21T02:13:16.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
82592025-03-21T02:13:16.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82602025-03-21T02:13:16.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82612025-03-21T02:13:16.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82622025-03-21T02:13:16.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82632025-03-21T02:13:16.934ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82642025-03-21T02:13:16.937ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
82652025-03-21T02:13:16.937ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82662025-03-21T02:13:16.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82672025-03-21T02:13:16.948ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
82682025-03-21T02:13:16.948ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.03816ms iteration = 54 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
82692025-03-21T02:13:19.722ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 292 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 7d944a4c-b52b-4064-8365-6873020c855a response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
82702025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
82712025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 19 reason = Timeout
82722025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 19 reason = Timeout
82732025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
82742025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 19 reason = Timeout
82752025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
82762025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 19 reason = Timeout
82772025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
82782025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
82792025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 10 reason = Timeout
82802025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
82812025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
82822025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
82832025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
82842025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 10 reason = Timeout
82852025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
82862025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 55 reason = Timeout
82872025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82882025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
82892025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
82902025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 19 reason = Timeout
82912025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
82922025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 10 reason = Timeout
82932025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
82942025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 10 reason = Timeout
82952025-03-21T02:13:26.921ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
82962025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
82972025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
82982025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
82992025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 19 reason = Timeout
83002025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
83012025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
83022025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
83032025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 19 reason = Timeout
83042025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
83052025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
83062025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 20 reason = Timeout
83072025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83082025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
83092025-03-21T02:13:26.921ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
83102025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.291\u{b5}s iteration = 11 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
83112025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
83122025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 19 reason = Timeout
83132025-03-21T02:13:26.921ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83142025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83152025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 19 reason = Timeout
83162025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 19 reason = Timeout
83172025-03-21T02:13:26.922ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
83182025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
83192025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
83202025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 5 reason = Timeout
83212025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83222025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
83232025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
83242025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
83252025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
83262025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 19 reason = Timeout
83272025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
83282025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
83292025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
83302025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
83312025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
83322025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
83332025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83342025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
83352025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 10 reason = Timeout
83362025-03-21T02:13:26.922ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 02:03:26.921930069 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
83372025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
83382025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
83392025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
83402025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
83412025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
83422025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
83432025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
83442025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
83452025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
83462025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
83472025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
83482025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
83492025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
83502025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
83512025-03-21T02:13:26.922ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1515, tv_nsec: 542813743 } })
83522025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
83532025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
83542025-03-21T02:13:26.922ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1515, tv_nsec: 542830743 } })
83552025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
83562025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83572025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
83582025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83592025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83602025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
83612025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
83622025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
83632025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83642025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
83652025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
83662025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
83672025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83682025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
83692025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
83702025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
83712025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
83722025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
83732025-03-21T02:13:26.922ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.188661ms iteration = 10 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
83742025-03-21T02:13:26.923ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
83752025-03-21T02:13:26.923ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.366604ms iteration = 10 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
83762025-03-21T02:13:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
83772025-03-21T02:13:26.923ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
83782025-03-21T02:13:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
83792025-03-21T02:13:26.925ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83802025-03-21T02:13:26.925ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1515, tv_nsec: 546285473 } })
83812025-03-21T02:13:26.925ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.490158ms iteration = 19 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
83822025-03-21T02:13:26.926ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
83832025-03-21T02:13:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
83842025-03-21T02:13:26.928ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
83852025-03-21T02:13:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
83862025-03-21T02:13:26.933ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 11.410489ms iteration = 10 status = {"errors":[],"found_port_ok":[]}
83872025-03-21T02:13:26.933ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83882025-03-21T02:13:26.933ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
83892025-03-21T02:13:26.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
83902025-03-21T02:13:26.934ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1515, tv_nsec: 554895183 } })
83912025-03-21T02:13:26.934ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
83922025-03-21T02:13:26.935ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
83932025-03-21T02:13:26.935ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
83942025-03-21T02:13:26.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
83952025-03-21T02:13:26.941ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83962025-03-21T02:13:26.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
83972025-03-21T02:13:26.945ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
83982025-03-21T02:13:26.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
83992025-03-21T02:13:26.947ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
84002025-03-21T02:13:26.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
84012025-03-21T02:13:26.950ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 02:03:26.921930069 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
84022025-03-21T02:13:26.950ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 28.870613ms iteration = 10 status = {"expiration":"2025-03-21T02:03:26.921930069Z","pruned":[]}
84032025-03-21T02:13:26.971ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 50.347416ms iteration = 19 status = {"errors":[],"garbage_collect_requested":[]}
84042025-03-21T02:13:26.972ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 50.819355ms iteration = 19 status = {"errors":[],"finish_invoked_ok":[]}
84052025-03-21T02:13:26.976ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
84062025-03-21T02:13:26.976ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84072025-03-21T02:13:26.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
84082025-03-21T02:13:26.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
84092025-03-21T02:13:26.979ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
84102025-03-21T02:13:26.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
84112025-03-21T02:13:26.980ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84122025-03-21T02:13:26.981ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.955294ms iteration = 55 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
84132025-03-21T02:13:26.982ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
84142025-03-21T02:13:26.982ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
84152025-03-21T02:13:26.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
84162025-03-21T02:13:26.985ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84172025-03-21T02:13:26.987ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
84182025-03-21T02:13:26.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
84192025-03-21T02:13:26.988ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84202025-03-21T02:13:26.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
84212025-03-21T02:13:26.988ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
84222025-03-21T02:13:26.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
84232025-03-21T02:13:26.989ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
84242025-03-21T02:13:26.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
84252025-03-21T02:13:26.989ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 68.057475ms iteration = 19 status = {"errors":[],"requests_created_ok":[]}
84262025-03-21T02:13:26.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
84272025-03-21T02:13:26.991ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84282025-03-21T02:13:26.991ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84292025-03-21T02:13:26.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84302025-03-21T02:13:26.991ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
84312025-03-21T02:13:26.991ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
84322025-03-21T02:13:26.991ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 70.187752ms iteration = 10 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
84332025-03-21T02:13:26.993ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
84342025-03-21T02:13:26.993ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84352025-03-21T02:13:26.993ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84362025-03-21T02:13:26.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84372025-03-21T02:13:26.994ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
84382025-03-21T02:13:26.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
84392025-03-21T02:13:26.994ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84402025-03-21T02:13:26.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
84412025-03-21T02:13:26.997ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84422025-03-21T02:13:26.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
84432025-03-21T02:13:26.999ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
84442025-03-21T02:13:26.999ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
84452025-03-21T02:13:26.999ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
84462025-03-21T02:13:27.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
84472025-03-21T02:13:27.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
84482025-03-21T02:13:27.001ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84492025-03-21T02:13:27.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84502025-03-21T02:13:27.002ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84512025-03-21T02:13:27.003ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84522025-03-21T02:13:27.004ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1515, tv_nsec: 625052924 } })
84532025-03-21T02:13:27.005ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
84542025-03-21T02:13:27.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
84552025-03-21T02:13:27.005ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
84562025-03-21T02:13:27.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
84572025-03-21T02:13:27.006ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
84582025-03-21T02:13:27.006ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84592025-03-21T02:13:27.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84602025-03-21T02:13:27.010ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84612025-03-21T02:13:27.010ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84622025-03-21T02:13:27.011ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84632025-03-21T02:13:27.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84642025-03-21T02:13:27.013ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
84652025-03-21T02:13:27.013ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
84662025-03-21T02:13:27.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
84672025-03-21T02:13:27.014ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
84682025-03-21T02:13:27.014ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84692025-03-21T02:13:27.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84702025-03-21T02:13:27.016ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
84712025-03-21T02:13:27.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
84722025-03-21T02:13:27.017ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
84732025-03-21T02:13:27.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
84742025-03-21T02:13:27.018ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84752025-03-21T02:13:27.018ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84762025-03-21T02:13:27.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84772025-03-21T02:13:27.020ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84782025-03-21T02:13:27.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
84792025-03-21T02:13:27.021ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
84802025-03-21T02:13:27.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
84812025-03-21T02:13:27.022ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84822025-03-21T02:13:27.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
84832025-03-21T02:13:27.024ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
84842025-03-21T02:13:27.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
84852025-03-21T02:13:27.025ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
84862025-03-21T02:13:27.025ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
84872025-03-21T02:13:27.025ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
84882025-03-21T02:13:27.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
84892025-03-21T02:13:27.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
84902025-03-21T02:13:27.026ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84912025-03-21T02:13:27.029ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
84922025-03-21T02:13:27.029ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84932025-03-21T02:13:27.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84942025-03-21T02:13:27.030ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84952025-03-21T02:13:27.032ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
84962025-03-21T02:13:27.032ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
84972025-03-21T02:13:27.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
84982025-03-21T02:13:27.034ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 112.674811ms iteration = 5 status = {"nsuccess":5}
84992025-03-21T02:13:27.038ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
85002025-03-21T02:13:27.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
85012025-03-21T02:13:27.039ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
85022025-03-21T02:13:27.039ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 117.580237ms iteration = 55 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
85032025-03-21T02:13:27.041ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
85042025-03-21T02:13:27.041ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 119.797996ms iteration = 19 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
85052025-03-21T02:13:27.043ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85062025-03-21T02:13:27.045ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
85072025-03-21T02:13:27.045ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
85082025-03-21T02:13:27.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
85092025-03-21T02:13:27.047ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
85102025-03-21T02:13:27.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
85112025-03-21T02:13:27.048ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
85122025-03-21T02:13:27.049ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 127.470959ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
85132025-03-21T02:13:27.049ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
85142025-03-21T02:13:27.049ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
85152025-03-21T02:13:27.049ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 127.818856ms iteration = 10 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
85162025-03-21T02:13:27.050ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
85172025-03-21T02:13:27.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
85182025-03-21T02:13:27.052ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
85192025-03-21T02:13:27.052ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
85202025-03-21T02:13:27.052ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
85212025-03-21T02:13:27.052ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
85222025-03-21T02:13:27.052ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
85232025-03-21T02:13:27.052ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
85242025-03-21T02:13:27.052ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
85252025-03-21T02:13:27.052ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
85262025-03-21T02:13:27.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
85272025-03-21T02:13:27.053ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ee2eb5b-2b38-4c3d-bbba-617b2fd1656d", "content-length": "26", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} })
85282025-03-21T02:13:27.053ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
85292025-03-21T02:13:27.054ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85302025-03-21T02:13:27.055ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
85312025-03-21T02:13:27.055ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
85322025-03-21T02:13:27.055ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
85332025-03-21T02:13:27.055ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
85342025-03-21T02:13:27.055ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
85352025-03-21T02:13:27.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
85362025-03-21T02:13:27.057ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85372025-03-21T02:13:27.057ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
85382025-03-21T02:13:27.057ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
85392025-03-21T02:13:27.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
85402025-03-21T02:13:27.060ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 138.982169ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
85412025-03-21T02:13:27.061ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
85422025-03-21T02:13:27.061ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
85432025-03-21T02:13:27.061ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
85442025-03-21T02:13:27.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
85452025-03-21T02:13:27.065ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 143.232614ms iteration = 19 status = 0
85462025-03-21T02:13:27.072ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
85472025-03-21T02:13:27.072ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
85482025-03-21T02:13:27.072ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
85492025-03-21T02:13:27.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
85502025-03-21T02:13:27.075ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
85512025-03-21T02:13:27.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
85522025-03-21T02:13:27.076ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
85532025-03-21T02:13:27.076ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
85542025-03-21T02:13:27.076ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2283cbea-438f-4df9-a469-22fdef7b1da9", "content-length": "26", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} })
85552025-03-21T02:13:27.076ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
85562025-03-21T02:13:27.077ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85572025-03-21T02:13:27.078ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
85582025-03-21T02:13:27.080ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
85592025-03-21T02:13:27.080ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
85602025-03-21T02:13:27.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
85612025-03-21T02:13:27.085ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 163.70035ms iteration = 20 status = {"physical_disks_added":0}
85622025-03-21T02:13:27.091ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 169.819926ms iteration = 19 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
85632025-03-21T02:13:27.092ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
85642025-03-21T02:13:27.092ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 170.989147ms iteration = 19 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
85652025-03-21T02:13:27.101ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
85662025-03-21T02:13:27.101ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
85672025-03-21T02:13:27.101ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
85682025-03-21T02:13:27.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
85692025-03-21T02:13:27.103ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85702025-03-21T02:13:27.103ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
85712025-03-21T02:13:27.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
85722025-03-21T02:13:27.107ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
85732025-03-21T02:13:27.107ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
85742025-03-21T02:13:27.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
85752025-03-21T02:13:27.109ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
85762025-03-21T02:13:27.109ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
85772025-03-21T02:13:27.109ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-21 02:04:31.662541 UTC dns_group = external generation = 2
85782025-03-21T02:13:27.109ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 187.336191ms iteration = 10 status = {"generation":2}
85792025-03-21T02:13:27.110ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
85802025-03-21T02:13:27.110ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
85812025-03-21T02:13:27.110ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 188.246687ms iteration = 19 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
85822025-03-21T02:13:27.112ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85832025-03-21T02:13:27.115ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
85842025-03-21T02:13:27.115ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
85852025-03-21T02:13:27.115ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
85862025-03-21T02:13:27.117ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) }
85872025-03-21T02:13:27.117ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 195.83069ms iteration = 10 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
85882025-03-21T02:13:27.166ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
85892025-03-21T02:13:27.166ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
85902025-03-21T02:13:27.166ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
85912025-03-21T02:13:27.167ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
85922025-03-21T02:13:27.168ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85932025-03-21T02:13:27.169ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
85942025-03-21T02:13:27.169ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 247.986638ms iteration = 10 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
85952025-03-21T02:13:27.171ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
85962025-03-21T02:13:27.171ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
85972025-03-21T02:13:27.171ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
85982025-03-21T02:13:27.174ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
85992025-03-21T02:13:27.174ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
86002025-03-21T02:13:27.174ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-21 02:04:30.886360 UTC dns_group = internal generation = 1
86012025-03-21T02:13:27.174ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 252.586097ms iteration = 10 status = {"generation":1}
86022025-03-21T02:13:27.206ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
86032025-03-21T02:13:27.207ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
86042025-03-21T02:13:27.208ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86052025-03-21T02:13:27.210ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
86062025-03-21T02:13:27.210ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
86072025-03-21T02:13:27.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
86082025-03-21T02:13:27.211ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86092025-03-21T02:13:27.213ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
86102025-03-21T02:13:27.213ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
86112025-03-21T02:13:27.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
86122025-03-21T02:13:27.215ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
86132025-03-21T02:13:27.215ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
86142025-03-21T02:13:27.216ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "34690683-34b9-48dc-8e72-0a30fc2d8386", "content-length": "26", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} })
86152025-03-21T02:13:27.216ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
86162025-03-21T02:13:27.218ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
86172025-03-21T02:13:27.253ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
86182025-03-21T02:13:27.253ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "bdecc356-27b2-4269-ad03-2461ffbb9af0", "date": "Fri, 21 Mar 2025 02:13:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
86192025-03-21T02:13:27.253ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
86202025-03-21T02:13:27.255ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "0f660888-2c3a-4dc1-8118-544dbddcab05", "date": "Fri, 21 Mar 2025 02:13:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
86212025-03-21T02:13:27.255ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
86222025-03-21T02:13:27.256ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
86232025-03-21T02:13:27.256ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 334.992092ms iteration = 12 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
86242025-03-21T02:13:27.258ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 337.22507ms iteration = 19 status = {}
86252025-03-21T02:13:27.326ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
86262025-03-21T02:13:27.327ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
86272025-03-21T02:13:27.327ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
86282025-03-21T02:13:27.346ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
86292025-03-21T02:13:27.347ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fa04374-47c5-4657-b04c-602ead6dce91", "content-length": "1", "date": "Fri, 21 Mar 2025 02:13:26 GMT"} })
86302025-03-21T02:13:27.348ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
86312025-03-21T02:13:27.349ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
86322025-03-21T02:13:27.351ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 429.355935ms iteration = 19 status = null
86332025-03-21T02:13:27.369ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "16a4c6c0-216c-46db-a015-1ce887f8ea73", "content-length": "372", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} })
86342025-03-21T02:13:27.369ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
86352025-03-21T02:13:27.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
86362025-03-21T02:13:27.384ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
86372025-03-21T02:13:27.384ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
86382025-03-21T02:13:27.385ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "8d3099ae-0feb-4d83-909f-35761f940426", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} })
86392025-03-21T02:13:27.385ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
86402025-03-21T02:13:27.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
86412025-03-21T02:13:27.387ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
86422025-03-21T02:13:27.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
86432025-03-21T02:13:27.405ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
86442025-03-21T02:13:27.405ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "35b4c1c0-728b-424d-8c45-3ecdfd8fee3e", "content-length": "691", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} })
86452025-03-21T02:13:27.405ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
86462025-03-21T02:13:27.405ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
86472025-03-21T02:13:27.405ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
86482025-03-21T02:13:27.406ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 484.514205ms iteration = 19 status = {}
86492025-03-21T02:13:27.408ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
86502025-03-21T02:13:27.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
86512025-03-21T02:13:27.411ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
86522025-03-21T02:13:27.411ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
86532025-03-21T02:13:27.411ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "650cc848-ae76-4b16-9f6a-a325518c0d4a", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} })
86542025-03-21T02:13:27.413ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 491.7121ms iteration = 19 status = {}
86552025-03-21T02:13:27.421ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
86562025-03-21T02:13:27.422ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "d8719ab0-799e-4bc6-a3dc-b4dcd464f9f9", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
86572025-03-21T02:13:27.422ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
86582025-03-21T02:13:27.424ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "5dfca047-41c5-4e67-9b77-1d96933baad8", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
86592025-03-21T02:13:27.424ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
86602025-03-21T02:13:27.425ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "c99bf0e0-f7e2-4fec-a0f2-e5f6a2f8b274", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
86612025-03-21T02:13:27.425ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
86622025-03-21T02:13:27.427ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
86632025-03-21T02:13:27.427ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 505.959648ms iteration = 12 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
86642025-03-21T02:13:27.457ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86652025-03-21T02:13:27.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86662025-03-21T02:13:27.458ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
86672025-03-21T02:13:27.458ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86682025-03-21T02:13:27.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86692025-03-21T02:13:27.469ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
86702025-03-21T02:13:27.470ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "669e7c66-97cc-4a41-8747-5d62b6887053", "content-length": "134", "date": "Fri, 21 Mar 2025 02:13:26 GMT"} })
86712025-03-21T02:13:27.470ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
86722025-03-21T02:13:27.470ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
86732025-03-21T02:13:27.470ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
86742025-03-21T02:13:27.470ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
86752025-03-21T02:13:27.470ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
86762025-03-21T02:13:27.470ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
86772025-03-21T02:13:27.470ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
86782025-03-21T02:13:27.470ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
86792025-03-21T02:13:27.470ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
86802025-03-21T02:13:27.470ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
86812025-03-21T02:13:27.471ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
86822025-03-21T02:13:27.471ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86832025-03-21T02:13:27.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86842025-03-21T02:13:27.472ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
86852025-03-21T02:13:27.472ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee953f62-8ca5-4562-9973-e8b9acea7b30", "content-length": "2", "date": "Fri, 21 Mar 2025 02:13:26 GMT"} })
86862025-03-21T02:13:27.472ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
86872025-03-21T02:13:27.472ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "36ad22fb-fafd-49db-a5d0-c2f49d0dc874", "content-length": "52", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} })
86882025-03-21T02:13:27.472ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
86892025-03-21T02:13:27.472ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
86902025-03-21T02:13:27.568ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "5e46305a-615f-4596-8cba-290b72cdb2d1", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} })
86912025-03-21T02:13:27.568ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
86922025-03-21T02:13:27.568ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
86932025-03-21T02:13:27.568ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f87f3e53-2577-4505-874e-1bf8bbc6637f", "content-length": "682", "date": "Fri, 21 Mar 2025 02:13:27 GMT"} })
86942025-03-21T02:13:27.568ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86952025-03-21T02:13:27.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86962025-03-21T02:13:27.572ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86972025-03-21T02:13:27.572ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86982025-03-21T02:13:27.576ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86992025-03-21T02:13:27.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
87002025-03-21T02:13:27.578ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
87012025-03-21T02:13:27.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
87022025-03-21T02:13:27.585ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 663.958579ms iteration = 19 status = {}
87032025-03-21T02:13:29.722ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 302 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 0fd1aa95-55d2-499e-a003-c4d78da92c95 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
87042025-03-21T02:13:36.921ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 56 reason = Timeout
87052025-03-21T02:13:36.921ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
87062025-03-21T02:13:36.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
87072025-03-21T02:13:36.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87082025-03-21T02:13:36.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
87092025-03-21T02:13:36.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87102025-03-21T02:13:36.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
87112025-03-21T02:13:36.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87122025-03-21T02:13:36.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
87132025-03-21T02:13:36.926ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
87142025-03-21T02:13:36.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87152025-03-21T02:13:36.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87162025-03-21T02:13:36.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87172025-03-21T02:13:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87182025-03-21T02:13:36.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87192025-03-21T02:13:36.932ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
87202025-03-21T02:13:36.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87212025-03-21T02:13:36.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87222025-03-21T02:13:36.941ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
87232025-03-21T02:13:36.941ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.838465ms iteration = 56 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
87242025-03-21T02:13:36.954ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.688159ms iteration = 56 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
87252025-03-21T02:13:39.722ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 294 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 97cad137-001a-4678-a20e-d974df9b00d6 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
87262025-03-21T02:13:46.921ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 57 reason = Timeout
87272025-03-21T02:13:46.921ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
87282025-03-21T02:13:46.921ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
87292025-03-21T02:13:46.921ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87302025-03-21T02:13:46.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87312025-03-21T02:13:46.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
87322025-03-21T02:13:46.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87332025-03-21T02:13:46.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
87342025-03-21T02:13:46.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
87352025-03-21T02:13:46.925ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.04768ms iteration = 57 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
87362025-03-21T02:13:46.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
87372025-03-21T02:13:46.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87382025-03-21T02:13:46.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87392025-03-21T02:13:46.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87402025-03-21T02:13:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87412025-03-21T02:13:46.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87422025-03-21T02:13:46.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
87432025-03-21T02:13:46.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87442025-03-21T02:13:46.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87452025-03-21T02:13:46.939ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
87462025-03-21T02:13:46.939ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.859851ms iteration = 57 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
87472025-03-21T02:13:49.722ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 296 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 52e40b5c-88c9-47fa-af94-b63068211467 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
87482025-03-21T02:13:54.713ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:35039
87492025-03-21T02:13:54.713ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35039 req_id = 63def168-fc5a-4a1b-83ad-31a86f58162d roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
87502025-03-21T02:13:54.713ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35039 req_id = 63def168-fc5a-4a1b-83ad-31a86f58162d resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
87512025-03-21T02:13:54.716ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3148 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35039 req_id = 63def168-fc5a-4a1b-83ad-31a86f58162d response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
87522025-03-21T02:13:54.716ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35039 req_id = 95bc1633-a85b-4682-8181-321ea31fc8ff roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
87532025-03-21T02:13:54.717ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35039 req_id = 95bc1633-a85b-4682-8181-321ea31fc8ff resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
87542025-03-21T02:13:54.719ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2228 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35039 req_id = 95bc1633-a85b-4682-8181-321ea31fc8ff response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
87552025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 58 reason = Timeout
87562025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
87572025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
87582025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
87592025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 58 reason = Timeout
87602025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 20 reason = Timeout
87612025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
87622025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87632025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87642025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 20 reason = Timeout
87652025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 20 reason = Timeout
87662025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
87672025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
87682025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 20 reason = Timeout
87692025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
87702025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
87712025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
87722025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
87732025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
87742025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
87752025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
87762025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 20 reason = Timeout
87772025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87782025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 20 reason = Timeout
87792025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
87802025-03-21T02:13:56.922ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
87812025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
87822025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 20 reason = Timeout
87832025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
87842025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
87852025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
87862025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 20 reason = Timeout
87872025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
87882025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 21 reason = Timeout
87892025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
87902025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 20 reason = Timeout
87912025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
87922025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 20 reason = Timeout
87932025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
87942025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
87952025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87962025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
87972025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87982025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
87992025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
88002025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
88012025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
88022025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
88032025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
88042025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
88052025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
88062025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
88072025-03-21T02:13:56.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
88082025-03-21T02:13:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88092025-03-21T02:13:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
88102025-03-21T02:13:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88112025-03-21T02:13:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
88122025-03-21T02:13:56.924ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.212848ms iteration = 20 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
88132025-03-21T02:13:56.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88142025-03-21T02:13:56.926ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88152025-03-21T02:13:56.927ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1545, tv_nsec: 547225413 } })
88162025-03-21T02:13:56.927ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
88172025-03-21T02:13:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
88182025-03-21T02:13:56.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88192025-03-21T02:13:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88202025-03-21T02:13:56.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
88212025-03-21T02:13:56.929ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88222025-03-21T02:13:56.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88232025-03-21T02:13:56.930ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1545, tv_nsec: 550933528 } })
88242025-03-21T02:13:56.931ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
88252025-03-21T02:13:56.931ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88262025-03-21T02:13:56.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
88272025-03-21T02:13:56.934ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
88282025-03-21T02:13:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
88292025-03-21T02:13:56.936ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88302025-03-21T02:13:56.936ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88312025-03-21T02:13:56.938ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88322025-03-21T02:13:56.938ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88332025-03-21T02:13:56.939ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
88342025-03-21T02:13:56.939ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
88352025-03-21T02:13:56.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
88362025-03-21T02:13:56.939ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
88372025-03-21T02:13:56.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88382025-03-21T02:13:56.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88392025-03-21T02:13:56.940ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
88402025-03-21T02:13:56.940ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
88412025-03-21T02:13:56.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
88422025-03-21T02:13:56.941ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
88432025-03-21T02:13:56.941ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88442025-03-21T02:13:56.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88452025-03-21T02:13:56.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88462025-03-21T02:13:56.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88472025-03-21T02:13:56.943ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88482025-03-21T02:13:56.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88492025-03-21T02:13:56.947ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
88502025-03-21T02:13:56.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
88512025-03-21T02:13:56.948ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1545, tv_nsec: 568991732 } })
88522025-03-21T02:13:56.949ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
88532025-03-21T02:13:56.950ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88542025-03-21T02:13:56.950ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88552025-03-21T02:13:56.952ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
88562025-03-21T02:13:56.952ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88572025-03-21T02:13:56.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88582025-03-21T02:13:56.953ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
88592025-03-21T02:13:56.953ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
88602025-03-21T02:13:56.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
88612025-03-21T02:13:56.967ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
88622025-03-21T02:13:56.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
88632025-03-21T02:13:56.975ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 52.951632ms iteration = 20 status = {"errors":[],"garbage_collect_requested":[]}
88642025-03-21T02:13:56.975ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
88652025-03-21T02:13:56.975ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
88662025-03-21T02:13:56.975ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
88672025-03-21T02:13:56.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
88682025-03-21T02:13:56.976ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 54.01421ms iteration = 20 status = {"errors":[],"requests_created_ok":[]}
88692025-03-21T02:13:56.978ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
88702025-03-21T02:13:56.979ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
88712025-03-21T02:13:56.979ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
88722025-03-21T02:13:56.979ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
88732025-03-21T02:13:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
88742025-03-21T02:13:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
88752025-03-21T02:13:56.981ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 58.528809ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
88762025-03-21T02:13:56.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
88772025-03-21T02:13:56.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
88782025-03-21T02:13:56.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
88792025-03-21T02:13:56.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
88802025-03-21T02:13:56.984ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 61.900558ms iteration = 20 status = {"errors":[],"finish_invoked_ok":[]}
88812025-03-21T02:13:56.984ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
88822025-03-21T02:13:56.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
88832025-03-21T02:13:56.991ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 68.88885ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
88842025-03-21T02:13:56.995ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88852025-03-21T02:13:56.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88862025-03-21T02:13:56.997ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88872025-03-21T02:13:56.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88882025-03-21T02:13:56.998ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
88892025-03-21T02:13:56.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
88902025-03-21T02:13:56.999ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88912025-03-21T02:13:57.001ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88922025-03-21T02:13:57.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
88932025-03-21T02:13:57.002ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88942025-03-21T02:13:57.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88952025-03-21T02:13:57.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
88962025-03-21T02:13:57.003ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88972025-03-21T02:13:57.003ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
88982025-03-21T02:13:57.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
88992025-03-21T02:13:57.004ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
89002025-03-21T02:13:57.004ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 82.159051ms iteration = 20 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
89012025-03-21T02:13:57.004ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 82.232792ms iteration = 20 status = 0
89022025-03-21T02:13:57.006ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
89032025-03-21T02:13:57.006ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 84.254627ms iteration = 20 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
89042025-03-21T02:13:57.007ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
89052025-03-21T02:13:57.007ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89062025-03-21T02:13:57.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89072025-03-21T02:13:57.009ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 87.693856ms iteration = 58 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
89082025-03-21T02:13:57.015ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
89092025-03-21T02:13:57.015ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
89102025-03-21T02:13:57.015ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1390cc0b-a741-4f62-8432-30d8be5124d3", "content-length": "26", "date": "Fri, 21 Mar 2025 02:13:56 GMT"} })
89112025-03-21T02:13:57.015ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
89122025-03-21T02:13:57.019ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
89132025-03-21T02:13:57.019ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
89142025-03-21T02:13:57.019ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "76121e99-f37b-424d-b00b-80198e3472ee", "content-length": "26", "date": "Fri, 21 Mar 2025 02:13:56 GMT"} })
89152025-03-21T02:13:57.019ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
89162025-03-21T02:13:57.020ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
89172025-03-21T02:13:57.020ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
89182025-03-21T02:13:57.020ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
89192025-03-21T02:13:57.020ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 98.087197ms iteration = 20 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
89202025-03-21T02:13:57.021ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
89212025-03-21T02:13:57.024ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
89222025-03-21T02:13:57.024ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 101.803293ms iteration = 58 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
89232025-03-21T02:13:57.028ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 105.516347ms iteration = 20 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
89242025-03-21T02:13:57.058ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
89252025-03-21T02:13:57.058ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
89262025-03-21T02:13:57.059ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f933d519-8255-4374-bd21-0a8588deee2b", "content-length": "26", "date": "Fri, 21 Mar 2025 02:13:57 GMT"} })
89272025-03-21T02:13:57.059ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
89282025-03-21T02:13:57.061ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
89292025-03-21T02:13:57.064ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 142.44296ms iteration = 21 status = {"physical_disks_added":0}
89302025-03-21T02:13:57.071ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
89312025-03-21T02:13:57.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
89322025-03-21T02:13:57.081ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
89332025-03-21T02:13:57.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
89342025-03-21T02:13:57.126ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
89352025-03-21T02:13:57.126ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "42fc2ff4-6a1f-49ac-b32d-2e7c9c8f2a30", "content-length": "1", "date": "Fri, 21 Mar 2025 02:13:57 GMT"} })
89362025-03-21T02:13:57.129ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
89372025-03-21T02:13:57.129ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
89382025-03-21T02:13:57.131ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 208.741654ms iteration = 20 status = {}
89392025-03-21T02:13:57.141ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
89402025-03-21T02:13:57.141ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
89412025-03-21T02:13:57.142ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b2a0add-c5a2-4b64-b34f-7b82bf94b5a5", "content-length": "372", "date": "Fri, 21 Mar 2025 02:13:56 GMT"} })
89422025-03-21T02:13:57.142ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89432025-03-21T02:13:57.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89442025-03-21T02:13:57.144ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 221.72631ms iteration = 20 status = null
89452025-03-21T02:13:57.157ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
89462025-03-21T02:13:57.157ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
89472025-03-21T02:13:57.157ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "416abfa1-788b-46d9-9e8f-da57dc85afe8", "date": "Fri, 21 Mar 2025 02:13:56 GMT"} })
89482025-03-21T02:13:57.157ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89492025-03-21T02:13:57.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89502025-03-21T02:13:57.160ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89512025-03-21T02:13:57.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89522025-03-21T02:13:57.163ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89532025-03-21T02:13:57.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89542025-03-21T02:13:57.167ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
89552025-03-21T02:13:57.167ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
89562025-03-21T02:13:57.167ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "ccab0104-695d-4414-b0ea-8c1079ee4147", "date": "Fri, 21 Mar 2025 02:13:56 GMT"} })
89572025-03-21T02:13:57.170ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 247.872955ms iteration = 20 status = {}
89582025-03-21T02:13:57.172ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89592025-03-21T02:13:57.174ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
89602025-03-21T02:13:57.174ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
89612025-03-21T02:13:57.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
89622025-03-21T02:13:57.175ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89632025-03-21T02:13:57.177ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
89642025-03-21T02:13:57.177ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
89652025-03-21T02:13:57.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
89662025-03-21T02:13:57.233ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
89672025-03-21T02:13:57.234ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e01047f-ec01-4b0b-81db-abea7a27fcfa", "content-length": "691", "date": "Fri, 21 Mar 2025 02:13:57 GMT"} })
89682025-03-21T02:13:57.234ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
89692025-03-21T02:13:57.234ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
89702025-03-21T02:13:57.234ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
89712025-03-21T02:13:57.234ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89722025-03-21T02:13:57.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89732025-03-21T02:13:57.235ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 313.160932ms iteration = 20 status = {}
89742025-03-21T02:13:57.237ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
89752025-03-21T02:13:57.237ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89762025-03-21T02:13:57.237ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89772025-03-21T02:13:57.246ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
89782025-03-21T02:13:57.247ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "54a9cae0-5196-4666-95af-377859a29cdc", "content-length": "134", "date": "Fri, 21 Mar 2025 02:13:57 GMT"} })
89792025-03-21T02:13:57.247ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
89802025-03-21T02:13:57.247ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
89812025-03-21T02:13:57.247ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
89822025-03-21T02:13:57.247ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
89832025-03-21T02:13:57.247ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89842025-03-21T02:13:57.247ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89852025-03-21T02:13:57.247ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89862025-03-21T02:13:57.247ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
89872025-03-21T02:13:57.247ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89882025-03-21T02:13:57.247ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
89892025-03-21T02:13:57.247ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
89902025-03-21T02:13:57.247ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89912025-03-21T02:13:57.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89922025-03-21T02:13:57.249ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
89932025-03-21T02:13:57.250ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6f7c052-cd85-41a6-8694-2160654a7443", "content-length": "2", "date": "Fri, 21 Mar 2025 02:13:57 GMT"} })
89942025-03-21T02:13:57.250ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
89952025-03-21T02:13:57.250ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "de787074-8449-49d4-8e45-dee4bf58f65d", "content-length": "52", "date": "Fri, 21 Mar 2025 02:13:57 GMT"} })
89962025-03-21T02:13:57.250ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
89972025-03-21T02:13:57.250ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
89982025-03-21T02:13:57.337ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "6c404589-a09b-4f5f-b497-d3d5bbd36ef8", "date": "Fri, 21 Mar 2025 02:13:57 GMT"} })
89992025-03-21T02:13:57.337ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
90002025-03-21T02:13:57.337ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
90012025-03-21T02:13:57.337ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "718d7fe7-64c2-4d69-9f15-ae2e5edd4b37", "content-length": "682", "date": "Fri, 21 Mar 2025 02:13:57 GMT"} })
90022025-03-21T02:13:57.337ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90032025-03-21T02:13:57.337ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90042025-03-21T02:13:57.341ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90052025-03-21T02:13:57.341ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90062025-03-21T02:13:57.344ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90072025-03-21T02:13:57.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90082025-03-21T02:13:57.346ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90092025-03-21T02:13:57.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90102025-03-21T02:13:57.353ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 431.085155ms iteration = 20 status = {}
90112025-03-21T02:13:59.722ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 277 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 5a8b12b9-2fa4-4b81-982b-c6d003b96a2a response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
90122025-03-21T02:14:06.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 59 reason = Timeout
90132025-03-21T02:14:06.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90142025-03-21T02:14:06.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 59 reason = Timeout
90152025-03-21T02:14:06.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90162025-03-21T02:14:06.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90172025-03-21T02:14:06.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90182025-03-21T02:14:06.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90192025-03-21T02:14:06.926ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
90202025-03-21T02:14:06.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90212025-03-21T02:14:06.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90222025-03-21T02:14:06.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90232025-03-21T02:14:06.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90242025-03-21T02:14:06.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90252025-03-21T02:14:06.932ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
90262025-03-21T02:14:06.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90272025-03-21T02:14:06.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90282025-03-21T02:14:06.941ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
90292025-03-21T02:14:06.941ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.49979ms iteration = 59 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
90302025-03-21T02:14:06.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90312025-03-21T02:14:06.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90322025-03-21T02:14:06.990ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 68.677945ms iteration = 59 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
90332025-03-21T02:14:09.723ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 300 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = c5b91a94-1631-47d9-b8ee-3760afd1c512 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
90342025-03-21T02:14:16.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 60 reason = Timeout
90352025-03-21T02:14:16.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 60 reason = Timeout
90362025-03-21T02:14:16.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90372025-03-21T02:14:16.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90382025-03-21T02:14:16.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90392025-03-21T02:14:16.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90402025-03-21T02:14:16.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90412025-03-21T02:14:16.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90422025-03-21T02:14:16.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90432025-03-21T02:14:16.926ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.952668ms iteration = 60 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
90442025-03-21T02:14:16.926ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
90452025-03-21T02:14:16.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90462025-03-21T02:14:16.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90472025-03-21T02:14:16.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90482025-03-21T02:14:16.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90492025-03-21T02:14:16.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90502025-03-21T02:14:16.932ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
90512025-03-21T02:14:16.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90522025-03-21T02:14:16.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90532025-03-21T02:14:16.938ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
90542025-03-21T02:14:16.938ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.372935ms iteration = 60 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
90552025-03-21T02:14:19.722ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 294 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = cfbd8d19-60ab-460e-8f12-636d341a749e response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
90562025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 61 reason = Timeout
90572025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 21 reason = Timeout
90582025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
90592025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90602025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
90612025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 61 reason = Timeout
90622025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 11 reason = Timeout
90632025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90642025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
90652025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
90662025-03-21T02:14:26.922ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 02:04:26.922545744 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
90672025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 2 reason = Timeout
90682025-03-21T02:14:26.922ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
90692025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
90702025-03-21T02:14:26.922ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
90712025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 21 reason = Timeout
90722025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
90732025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 3 reason = Timeout
90742025-03-21T02:14:26.922ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
90752025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
90762025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 3 reason = Timeout
90772025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
90782025-03-21T02:14:26.922ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
90792025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
90802025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.67\u{b5}s iteration = 12 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
90812025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 21 reason = Timeout
90822025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 21 reason = Timeout
90832025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
90842025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
90852025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
90862025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Timeout
90872025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90882025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 11 reason = Timeout
90892025-03-21T02:14:26.922ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
90902025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
90912025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 21 reason = Timeout
90922025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
90932025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 21 reason = Timeout
90942025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
90952025-03-21T02:14:26.922ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
90962025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 21 reason = Timeout
90972025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
90982025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 21 reason = Timeout
90992025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
91002025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
91012025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 21 reason = Timeout
91022025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
91032025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 21 reason = Timeout
91042025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
91052025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 11 reason = Timeout
91062025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
91072025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 21 reason = Timeout
91082025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
91092025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 21 reason = Timeout
91102025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
91112025-03-21T02:14:26.923ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
91122025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 22 reason = Timeout
91132025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91142025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
91152025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
91162025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 21 reason = Timeout
91172025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
91182025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
91192025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
91202025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
91212025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
91222025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 21 reason = Timeout
91232025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
91242025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
91252025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 21 reason = Timeout
91262025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
91272025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
91282025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 21 reason = Timeout
91292025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
91302025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
91312025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
91322025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 4 reason = Timeout
91332025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
91342025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 11 reason = Timeout
91352025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
91362025-03-21T02:14:26.923ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
91372025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91382025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
91392025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
91402025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
91412025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
91422025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91432025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
91442025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
91452025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
91462025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
91472025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
91482025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
91492025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
91502025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
91512025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
91522025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
91532025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
91542025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91552025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
91562025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
91572025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
91582025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91592025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
91602025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
91612025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
91622025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
91632025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
91642025-03-21T02:14:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
91652025-03-21T02:14:26.923ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1575, tv_nsec: 543417209 } })
91662025-03-21T02:14:26.923ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1575, tv_nsec: 543423869 } })
91672025-03-21T02:14:26.924ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
91682025-03-21T02:14:26.924ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.418865ms iteration = 11 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
91692025-03-21T02:14:26.924ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
91702025-03-21T02:14:26.924ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.629139ms iteration = 11 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
91712025-03-21T02:14:26.925ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
91722025-03-21T02:14:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
91732025-03-21T02:14:26.927ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 02:04:26.922545744 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
91742025-03-21T02:14:26.927ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.060789ms iteration = 11 status = {"expiration":"2025-03-21T02:04:26.922545744Z","pruned":[]}
91752025-03-21T02:14:26.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91762025-03-21T02:14:26.928ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1575, tv_nsec: 548444927 } })
91772025-03-21T02:14:26.928ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.981444ms iteration = 21 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
91782025-03-21T02:14:26.928ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
91792025-03-21T02:14:26.929ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
91802025-03-21T02:14:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
91812025-03-21T02:14:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
91822025-03-21T02:14:26.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91832025-03-21T02:14:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91842025-03-21T02:14:26.933ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 10.957591ms iteration = 11 status = {"errors":[],"found_port_ok":[]}
91852025-03-21T02:14:26.933ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91862025-03-21T02:14:26.934ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
91872025-03-21T02:14:26.934ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91882025-03-21T02:14:26.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
91892025-03-21T02:14:26.934ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91902025-03-21T02:14:26.935ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91912025-03-21T02:14:26.937ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
91922025-03-21T02:14:26.937ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
91932025-03-21T02:14:26.937ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
91942025-03-21T02:14:26.937ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
91952025-03-21T02:14:26.937ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
91962025-03-21T02:14:26.937ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
91972025-03-21T02:14:26.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
91982025-03-21T02:14:26.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
91992025-03-21T02:14:26.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
92002025-03-21T02:14:26.938ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
92012025-03-21T02:14:26.938ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
92022025-03-21T02:14:26.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
92032025-03-21T02:14:26.939ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
92042025-03-21T02:14:26.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
92052025-03-21T02:14:26.941ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1575, tv_nsec: 561488954 } })
92062025-03-21T02:14:26.942ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
92072025-03-21T02:14:26.943ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92082025-03-21T02:14:26.945ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
92092025-03-21T02:14:26.945ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92102025-03-21T02:14:26.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92112025-03-21T02:14:26.946ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92122025-03-21T02:14:26.947ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
92132025-03-21T02:14:26.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
92142025-03-21T02:14:26.949ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
92152025-03-21T02:14:26.949ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
92162025-03-21T02:14:26.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
92172025-03-21T02:14:26.952ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92182025-03-21T02:14:26.953ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
92192025-03-21T02:14:26.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
92202025-03-21T02:14:26.954ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
92212025-03-21T02:14:26.954ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
92222025-03-21T02:14:26.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
92232025-03-21T02:14:26.959ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92242025-03-21T02:14:26.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92252025-03-21T02:14:26.960ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
92262025-03-21T02:14:26.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
92272025-03-21T02:14:26.963ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92282025-03-21T02:14:26.965ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
92292025-03-21T02:14:26.965ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
92302025-03-21T02:14:26.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
92312025-03-21T02:14:26.975ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
92322025-03-21T02:14:26.975ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 52.737768ms iteration = 2 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
92332025-03-21T02:14:26.978ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
92342025-03-21T02:14:26.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
92352025-03-21T02:14:26.981ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
92362025-03-21T02:14:26.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
92372025-03-21T02:14:26.983ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 61.122684ms iteration = 21 status = {"errors":[],"garbage_collect_requested":[]}
92382025-03-21T02:14:26.987ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92392025-03-21T02:14:26.987ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92402025-03-21T02:14:26.990ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92412025-03-21T02:14:26.990ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92422025-03-21T02:14:26.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92432025-03-21T02:14:26.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
92442025-03-21T02:14:26.991ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1575, tv_nsec: 611603116 } })
92452025-03-21T02:14:26.992ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
92462025-03-21T02:14:26.995ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 72.36029ms iteration = 21 status = {"errors":[],"requests_created_ok":[]}
92472025-03-21T02:14:26.995ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
92482025-03-21T02:14:26.995ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92492025-03-21T02:14:26.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92502025-03-21T02:14:26.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
92512025-03-21T02:14:26.995ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
92522025-03-21T02:14:26.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
92532025-03-21T02:14:26.997ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
92542025-03-21T02:14:26.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
92552025-03-21T02:14:26.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
92562025-03-21T02:14:27.000ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
92572025-03-21T02:14:27.000ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
92582025-03-21T02:14:27.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
92592025-03-21T02:14:27.003ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92602025-03-21T02:14:27.004ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92612025-03-21T02:14:27.004ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92622025-03-21T02:14:27.004ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92632025-03-21T02:14:27.004ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
92642025-03-21T02:14:27.004ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
92652025-03-21T02:14:27.004ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
92662025-03-21T02:14:27.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
92672025-03-21T02:14:27.005ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92682025-03-21T02:14:27.008ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
92692025-03-21T02:14:27.008ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
92702025-03-21T02:14:27.008ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
92712025-03-21T02:14:27.008ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
92722025-03-21T02:14:27.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
92732025-03-21T02:14:27.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
92742025-03-21T02:14:27.008ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
92752025-03-21T02:14:27.008ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
92762025-03-21T02:14:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
92772025-03-21T02:14:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
92782025-03-21T02:14:27.009ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92792025-03-21T02:14:27.009ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
92802025-03-21T02:14:27.009ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
92812025-03-21T02:14:27.009ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
92822025-03-21T02:14:27.009ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 86.398385ms iteration = 11 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
92832025-03-21T02:14:27.009ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
92842025-03-21T02:14:27.009ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
92852025-03-21T02:14:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92862025-03-21T02:14:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
92872025-03-21T02:14:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
92882025-03-21T02:14:27.010ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
92892025-03-21T02:14:27.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
92902025-03-21T02:14:27.011ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92912025-03-21T02:14:27.013ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 90.490116ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
92922025-03-21T02:14:27.014ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
92932025-03-21T02:14:27.014ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
92942025-03-21T02:14:27.014ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
92952025-03-21T02:14:27.014ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 91.663956ms iteration = 61 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
92962025-03-21T02:14:27.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
92972025-03-21T02:14:27.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
92982025-03-21T02:14:27.016ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
92992025-03-21T02:14:27.016ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
93002025-03-21T02:14:27.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
93012025-03-21T02:14:27.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
93022025-03-21T02:14:27.017ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
93032025-03-21T02:14:27.017ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
93042025-03-21T02:14:27.017ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
93052025-03-21T02:14:27.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
93062025-03-21T02:14:27.017ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93072025-03-21T02:14:27.017ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 95.103975ms iteration = 21 status = {"errors":[],"finish_invoked_ok":[]}
93082025-03-21T02:14:27.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93092025-03-21T02:14:27.023ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
93102025-03-21T02:14:27.023ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
93112025-03-21T02:14:27.023ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 101.216112ms iteration = 11 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
93122025-03-21T02:14:27.027ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
93132025-03-21T02:14:27.027ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 104.894916ms iteration = 61 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
93142025-03-21T02:14:27.032ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
93152025-03-21T02:14:27.032ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
93162025-03-21T02:14:27.032ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
93172025-03-21T02:14:27.032ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
93182025-03-21T02:14:27.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
93192025-03-21T02:14:27.035ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c619b06c-12f1-424e-9c3c-2bdf07aa5f54, 0), (1a813287-9754-40a2-b7fb-0cf070a070ed, 0), (2e917296-4190-4257-a809-df5c6fd8875a, 0)]
93202025-03-21T02:14:27.035ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
93212025-03-21T02:14:27.035ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 113.097399ms iteration = 21 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
93222025-03-21T02:14:27.036ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1575, tv_nsec: 656007689 } })
93232025-03-21T02:14:27.040ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
93242025-03-21T02:14:27.040ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
93252025-03-21T02:14:27.040ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
93262025-03-21T02:14:27.040ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
93272025-03-21T02:14:27.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
93282025-03-21T02:14:27.041ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
93292025-03-21T02:14:27.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
93302025-03-21T02:14:27.051ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 128.531188ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
93312025-03-21T02:14:27.051ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
93322025-03-21T02:14:27.065ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
93332025-03-21T02:14:27.077ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
93342025-03-21T02:14:27.077ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
93352025-03-21T02:14:27.078ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a4a362b-17fb-49c0-a5a9-eeb4ae6d6020", "content-length": "26", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
93362025-03-21T02:14:27.078ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
93372025-03-21T02:14:27.081ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
93382025-03-21T02:14:27.102ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
93392025-03-21T02:14:27.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
93402025-03-21T02:14:27.126ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 203.825809ms iteration = 22 status = {"physical_disks_added":0}
93412025-03-21T02:14:27.140ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
93422025-03-21T02:14:27.140ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
93432025-03-21T02:14:27.140ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "144ff59e-03aa-4fc8-addf-3f159ee8aa03", "content-length": "26", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
93442025-03-21T02:14:27.140ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
93452025-03-21T02:14:27.142ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
93462025-03-21T02:14:27.241ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
93472025-03-21T02:14:27.242ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
93482025-03-21T02:14:27.244ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93492025-03-21T02:14:27.245ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
93502025-03-21T02:14:27.245ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
93512025-03-21T02:14:27.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
93522025-03-21T02:14:27.249ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93532025-03-21T02:14:27.250ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93542025-03-21T02:14:27.250ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
93552025-03-21T02:14:27.251ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93562025-03-21T02:14:27.251ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
93572025-03-21T02:14:27.251ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "b09a53a1-be34-44e3-bbdf-088f3cff2497", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
93582025-03-21T02:14:27.251ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
93592025-03-21T02:14:27.251ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
93602025-03-21T02:14:27.251ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
93612025-03-21T02:14:27.252ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
93622025-03-21T02:14:27.253ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "639825a7-53ad-4f83-91f2-63ca0a2b714f", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
93632025-03-21T02:14:27.253ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
93642025-03-21T02:14:27.255ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
93652025-03-21T02:14:27.255ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
93662025-03-21T02:14:27.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
93672025-03-21T02:14:27.255ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
93682025-03-21T02:14:27.255ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93692025-03-21T02:14:27.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93702025-03-21T02:14:27.255ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
93712025-03-21T02:14:27.255ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 333.371604ms iteration = 13 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
93722025-03-21T02:14:27.257ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93732025-03-21T02:14:27.258ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93742025-03-21T02:14:27.260ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93752025-03-21T02:14:27.260ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
93762025-03-21T02:14:27.260ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
93772025-03-21T02:14:27.260ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93782025-03-21T02:14:27.260ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
93792025-03-21T02:14:27.260ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
93802025-03-21T02:14:27.260ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
93812025-03-21T02:14:27.261ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
93822025-03-21T02:14:27.266ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
93832025-03-21T02:14:27.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
93842025-03-21T02:14:27.266ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
93852025-03-21T02:14:27.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
93862025-03-21T02:14:27.274ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
93872025-03-21T02:14:27.274ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
93882025-03-21T02:14:27.274ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
93892025-03-21T02:14:27.275ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "af4b5935-450c-45f0-bf03-5eaf85162057", "content-length": "476", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
93902025-03-21T02:14:27.275ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
93912025-03-21T02:14:27.275ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39813425-bce9-4438-a367-a5cccacfc7a6", "content-length": "734", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
93922025-03-21T02:14:27.275ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
93932025-03-21T02:14:27.275ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "35f2fbd5-9e16-46c5-8c84-8caf50d789f1", "content-length": "111", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
93942025-03-21T02:14:27.275ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
93952025-03-21T02:14:27.276ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8605ad8f-4c64-4a2a-b908-bed2a8e9a0b3", "content-length": "111", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
93962025-03-21T02:14:27.276ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
93972025-03-21T02:14:27.276ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f67ac93-d07a-437a-bd3d-229d0be8ef6a", "content-length": "167", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
93982025-03-21T02:14:27.276ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
93992025-03-21T02:14:27.276ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db69445f-0535-4a58-b6d2-e1245cb90d7c", "content-length": "167", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
94002025-03-21T02:14:27.276ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
94012025-03-21T02:14:27.277ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76bc9873-c46e-4746-a2f9-c6d3f9dc9734", "content-length": "175", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
94022025-03-21T02:14:27.277ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
94032025-03-21T02:14:27.277ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "355204c2-12c7-43a9-9d00-cdf9382b9d19", "content-length": "175", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
94042025-03-21T02:14:27.277ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
94052025-03-21T02:14:27.277ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff377d46-1f91-473d-89b9-fa70cfcf4d7b", "content-length": "702", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
94062025-03-21T02:14:27.277ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
94072025-03-21T02:14:27.278ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0cf9287-b41d-48a5-a9f4-31ba7de0ce73", "content-length": "727", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94082025-03-21T02:14:27.278ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
94092025-03-21T02:14:27.278ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d7b913e-9b52-4f46-960e-ed9253a0b1ff", "content-length": "729", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94102025-03-21T02:14:27.278ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
94112025-03-21T02:14:27.278ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95759fe6-016d-45bf-95c4-8e3ddc05930d", "content-length": "728", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94122025-03-21T02:14:27.278ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
94132025-03-21T02:14:27.278ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "644bdb1e-5a02-4ead-ac94-0efdd237c5d1", "content-length": "722", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94142025-03-21T02:14:27.278ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
94152025-03-21T02:14:27.279ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51871a58-ba1f-44d9-9122-c67c26a6495c", "content-length": "109", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94162025-03-21T02:14:27.279ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
94172025-03-21T02:14:27.279ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c9ddb5c-b010-48a7-8c39-9d23d54e5d16", "content-length": "109", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94182025-03-21T02:14:27.279ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
94192025-03-21T02:14:27.279ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 356.889584ms iteration = 21 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
94202025-03-21T02:14:27.279ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd74ec62-85ba-4e29-9a3f-06ea3ed454b9", "content-length": "169", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94212025-03-21T02:14:27.279ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
94222025-03-21T02:14:27.280ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "611cdae0-2600-4406-b319-c6ac4f9e7ada", "content-length": "169", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94232025-03-21T02:14:27.280ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
94242025-03-21T02:14:27.280ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a851b34a-1ca8-44a9-a960-168f73085e63", "content-length": "178", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94252025-03-21T02:14:27.280ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
94262025-03-21T02:14:27.280ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1993a99-a76f-4f9c-a243-862de69b7817", "content-length": "178", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94272025-03-21T02:14:27.280ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
94282025-03-21T02:14:27.281ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d82c2068-5921-4f28-a790-6eab7467962f", "content-length": "702", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94292025-03-21T02:14:27.281ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
94302025-03-21T02:14:27.281ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6618991-cdd7-471a-b10d-b16f2d85090c", "content-length": "727", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94312025-03-21T02:14:27.281ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
94322025-03-21T02:14:27.281ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d283fb0e-cca4-4ead-bbd4-1fddd56c7a89", "content-length": "729", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
94332025-03-21T02:14:27.281ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
94342025-03-21T02:14:27.281ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "99178b84-52e9-4912-8100-98f27a9f4789", "content-length": "728", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
94352025-03-21T02:14:27.281ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94362025-03-21T02:14:27.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94372025-03-21T02:14:27.282ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
94382025-03-21T02:14:27.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
94392025-03-21T02:14:27.286ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
94402025-03-21T02:14:27.286ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 364.087688ms iteration = 21 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
94412025-03-21T02:14:27.294ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
94422025-03-21T02:14:27.294ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
94432025-03-21T02:14:27.294ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
94442025-03-21T02:14:27.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
94452025-03-21T02:14:27.297ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
94462025-03-21T02:14:27.297ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
94472025-03-21T02:14:27.297ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 374.337136ms iteration = 21 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
94482025-03-21T02:14:27.301ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
94492025-03-21T02:14:27.301ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
94502025-03-21T02:14:27.301ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7d0c2cf-db26-439b-8609-9f0d732994c7", "content-length": "26", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
94512025-03-21T02:14:27.301ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
94522025-03-21T02:14:27.303ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
94532025-03-21T02:14:27.322ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) }
94542025-03-21T02:14:27.322ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 399.298511ms iteration = 11 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
94552025-03-21T02:14:27.332ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 409.485388ms iteration = 21 status = {}
94562025-03-21T02:14:27.337ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
94572025-03-21T02:14:27.337ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 415.142668ms iteration = 11 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
94582025-03-21T02:14:27.352ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
94592025-03-21T02:14:27.353ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "19597e5e-939e-42d7-87aa-d0ba1af9bd82", "content-length": "1", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
94602025-03-21T02:14:27.354ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
94612025-03-21T02:14:27.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
94622025-03-21T02:14:27.379ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
94632025-03-21T02:14:27.379ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
94642025-03-21T02:14:27.379ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "81b99d83-622f-47e4-ab3e-301030ddf035", "content-length": "372", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
94652025-03-21T02:14:27.379ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
94662025-03-21T02:14:27.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
94672025-03-21T02:14:27.380ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94682025-03-21T02:14:27.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94692025-03-21T02:14:27.382ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
94702025-03-21T02:14:27.383ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
94712025-03-21T02:14:27.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
94722025-03-21T02:14:27.384ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94732025-03-21T02:14:27.384ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 462.045803ms iteration = 21 status = null
94742025-03-21T02:14:27.387ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94752025-03-21T02:14:27.388ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
94762025-03-21T02:14:27.388ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
94772025-03-21T02:14:27.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
94782025-03-21T02:14:27.394ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
94792025-03-21T02:14:27.394ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94802025-03-21T02:14:27.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94812025-03-21T02:14:27.396ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "5de6c67c-f94b-4257-a889-eafe0dde5dc3", "content-length": "2", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
94822025-03-21T02:14:27.396ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:615 sled = http://[fd00:1122:3344:101::1]:12345
94832025-03-21T02:14:27.397ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
94842025-03-21T02:14:27.397ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
94852025-03-21T02:14:27.397ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-21 02:04:30.886360 UTC dns_group = internal generation = 1
94862025-03-21T02:14:27.397ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 474.705204ms iteration = 11 status = {"generation":1}
94872025-03-21T02:14:27.398ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94882025-03-21T02:14:27.398ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94892025-03-21T02:14:27.398ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 476.235281ms iteration = 3 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":2,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"ad5b28f8-6cad-4221-97f9-e84a46a6381b","time":"2025-03-21T02:09:27.430025209Z"},{"operation":"list","target_sled":"ad5b28f8-6cad-4221-97f9-e84a46a6381b","time":"2025-03-21T02:14:27.396925352Z"}]}
94902025-03-21T02:14:27.400ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94912025-03-21T02:14:27.401ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94922025-03-21T02:14:27.403ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94932025-03-21T02:14:27.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94942025-03-21T02:14:27.405ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94952025-03-21T02:14:27.405ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94962025-03-21T02:14:27.407ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94972025-03-21T02:14:27.407ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94982025-03-21T02:14:27.412ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
94992025-03-21T02:14:27.413ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "7f4b4e86-498a-43e9-b501-a80135cfdc4e", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
95002025-03-21T02:14:27.413ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
95012025-03-21T02:14:27.415ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "f3c85287-7c92-451d-bb5b-d49e85659517", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
95022025-03-21T02:14:27.415ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
95032025-03-21T02:14:27.416ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
95042025-03-21T02:14:27.416ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
95052025-03-21T02:14:27.416ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
95062025-03-21T02:14:27.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
95072025-03-21T02:14:27.416ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "c8c2e59d-d2b1-474a-94d8-fc9c26b4e2f8", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
95082025-03-21T02:14:27.416ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
95092025-03-21T02:14:27.418ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95102025-03-21T02:14:27.418ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
95112025-03-21T02:14:27.418ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 495.245692ms iteration = 13 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
95122025-03-21T02:14:27.420ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
95132025-03-21T02:14:27.420ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
95142025-03-21T02:14:27.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
95152025-03-21T02:14:27.424ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
95162025-03-21T02:14:27.424ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
95172025-03-21T02:14:27.424ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-21 02:04:31.662541 UTC dns_group = external generation = 2
95182025-03-21T02:14:27.424ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 501.320497ms iteration = 11 status = {"generation":2}
95192025-03-21T02:14:27.427ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95202025-03-21T02:14:27.427ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95212025-03-21T02:14:27.429ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
95222025-03-21T02:14:27.429ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95232025-03-21T02:14:27.430ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95242025-03-21T02:14:27.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95252025-03-21T02:14:27.430ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
95262025-03-21T02:14:27.430ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
95272025-03-21T02:14:27.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
95282025-03-21T02:14:27.431ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95292025-03-21T02:14:27.431ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95302025-03-21T02:14:27.432ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
95312025-03-21T02:14:27.432ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
95322025-03-21T02:14:27.432ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
95332025-03-21T02:14:27.433ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 510.11175ms iteration = 21 status = 0
95342025-03-21T02:14:27.433ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
95352025-03-21T02:14:27.433ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
95362025-03-21T02:14:27.433ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
95372025-03-21T02:14:27.434ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
95382025-03-21T02:14:27.434ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95392025-03-21T02:14:27.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95402025-03-21T02:14:27.435ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95412025-03-21T02:14:27.436ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
95422025-03-21T02:14:27.436ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
95432025-03-21T02:14:27.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
95442025-03-21T02:14:27.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
95452025-03-21T02:14:27.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
95462025-03-21T02:14:27.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95472025-03-21T02:14:27.442ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
95482025-03-21T02:14:27.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
95492025-03-21T02:14:27.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
95502025-03-21T02:14:27.442ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
95512025-03-21T02:14:27.442ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
95522025-03-21T02:14:27.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95532025-03-21T02:14:27.445ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
95542025-03-21T02:14:27.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
95552025-03-21T02:14:27.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
95562025-03-21T02:14:27.445ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95572025-03-21T02:14:27.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95582025-03-21T02:14:27.448ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
95592025-03-21T02:14:27.448ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 525.283664ms iteration = 4 status = {"blueprint_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","inventory_collection_id":"2e917296-4190-4257-a809-df5c6fd8875a","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
95602025-03-21T02:14:27.455ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "54c1fa2e-b2e8-44ef-87e0-c30605d3209f", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
95612025-03-21T02:14:27.455ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95622025-03-21T02:14:27.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95632025-03-21T02:14:27.468ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95642025-03-21T02:14:27.470ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
95652025-03-21T02:14:27.470ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
95662025-03-21T02:14:27.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
95672025-03-21T02:14:27.501ZDEBGnexus (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[fd00:1122:3344:101::1]:12345
95682025-03-21T02:14:27.501ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/inventory
95692025-03-21T02:14:27.503ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95702025-03-21T02:14:27.504ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95712025-03-21T02:14:27.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95722025-03-21T02:14:27.507ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
95732025-03-21T02:14:27.507ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95742025-03-21T02:14:27.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95752025-03-21T02:14:27.511ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95762025-03-21T02:14:27.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95772025-03-21T02:14:27.512ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95782025-03-21T02:14:27.512ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95792025-03-21T02:14:27.514ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95802025-03-21T02:14:27.514ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
95812025-03-21T02:14:27.514ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
95822025-03-21T02:14:27.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95832025-03-21T02:14:27.518ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "06080432-5624-4544-8b1f-c30b938582c6", "date": "Fri, 21 Mar 2025 02:14:26 GMT"} })
95842025-03-21T02:14:27.520ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 597.426581ms iteration = 21 status = {}
95852025-03-21T02:14:27.561ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95862025-03-21T02:14:27.561ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95872025-03-21T02:14:27.566ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
95882025-03-21T02:14:27.567ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "7123ec3c-e359-4dd8-89ca-9a5a239db388", "content-length": "691", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
95892025-03-21T02:14:27.567ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
95902025-03-21T02:14:27.567ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
95912025-03-21T02:14:27.567ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
95922025-03-21T02:14:27.568ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 645.558418ms iteration = 21 status = {}
95932025-03-21T02:14:27.569ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95942025-03-21T02:14:27.569ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95952025-03-21T02:14:27.569ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
95962025-03-21T02:14:27.569ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95972025-03-21T02:14:27.569ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95982025-03-21T02:14:27.571ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95992025-03-21T02:14:27.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96002025-03-21T02:14:27.572ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96012025-03-21T02:14:27.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96022025-03-21T02:14:27.574ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96032025-03-21T02:14:27.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96042025-03-21T02:14:27.575ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96052025-03-21T02:14:27.579ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
96062025-03-21T02:14:27.579ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
96072025-03-21T02:14:27.579ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96082025-03-21T02:14:27.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96092025-03-21T02:14:27.579ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "549e05bb-fa2a-4b8d-96a2-1d12990dcabc", "content-length": "134", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
96102025-03-21T02:14:27.579ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
96112025-03-21T02:14:27.579ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
96122025-03-21T02:14:27.580ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
96132025-03-21T02:14:27.580ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
96142025-03-21T02:14:27.580ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
96152025-03-21T02:14:27.580ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
96162025-03-21T02:14:27.580ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
96172025-03-21T02:14:27.580ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
96182025-03-21T02:14:27.580ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
96192025-03-21T02:14:27.580ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
96202025-03-21T02:14:27.580ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
96212025-03-21T02:14:27.580ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96222025-03-21T02:14:27.580ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96232025-03-21T02:14:27.580ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
96242025-03-21T02:14:27.581ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96252025-03-21T02:14:27.582ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96262025-03-21T02:14:27.582ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
96272025-03-21T02:14:27.582ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96282025-03-21T02:14:27.582ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "41e82b83-5068-4351-b0f9-b4748f08e8a2", "content-length": "2", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
96292025-03-21T02:14:27.582ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
96302025-03-21T02:14:27.582ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f6d56eb-f1af-4ec7-aea9-84c3b730acc9", "content-length": "52", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
96312025-03-21T02:14:27.582ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
96322025-03-21T02:14:27.582ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
96332025-03-21T02:14:27.584ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96342025-03-21T02:14:27.587ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
96352025-03-21T02:14:27.587ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96362025-03-21T02:14:27.588ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96372025-03-21T02:14:27.592ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96382025-03-21T02:14:27.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96392025-03-21T02:14:27.594ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96402025-03-21T02:14:27.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96412025-03-21T02:14:27.595ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96422025-03-21T02:14:27.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96432025-03-21T02:14:27.597ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96442025-03-21T02:14:27.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96452025-03-21T02:14:27.600ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96462025-03-21T02:14:27.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96472025-03-21T02:14:27.601ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96482025-03-21T02:14:27.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96492025-03-21T02:14:27.603ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96502025-03-21T02:14:27.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96512025-03-21T02:14:27.604ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96522025-03-21T02:14:27.608ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
96532025-03-21T02:14:27.608ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96542025-03-21T02:14:27.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96552025-03-21T02:14:27.612ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96562025-03-21T02:14:27.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96572025-03-21T02:14:27.616ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96582025-03-21T02:14:27.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96592025-03-21T02:14:27.617ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96602025-03-21T02:14:27.621ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
96612025-03-21T02:14:27.621ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96622025-03-21T02:14:27.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96632025-03-21T02:14:27.628ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 34365a8b-47e5-4623-9ff4-fa54efca49f0, kind: Service { id: 6021c9d9-927c-44a6-8331-6ab48efd9c00 }, name: Name("nexus-6021c9d9-927c-44a6-8331-6ab48efd9c00"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 210])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8d8a5138-cad9-4d17-aa4a-561116e6b6ee, kind: Service { id: e05eb799-8cf0-4702-bd3b-9994ae86118a }, name: Name("nexus-e05eb799-8cf0-4702-bd3b-9994ae86118a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 112])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c4619e25-4546-4262-8518-a5e7bdb39205, kind: Service { id: 4b42f969-b761-4b99-9019-b41841cb0f60 }, name: Name("nexus-4b42f969-b761-4b99-9019-b41841cb0f60"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 128])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 5a5beddc-544e-4e62-bf20-c61d9dde2785, kind: Service { id: bfbad16c-d6a4-4742-9db2-139d724c0771 }, name: Name("external-dns-bfbad16c-d6a4-4742-9db2-139d724c0771"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 221])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bfb6f529-ad0a-4d8f-8e27-90ee0b561301, kind: Service { id: a1777e88-1a9c-49b1-bca1-ec6c5e1991fc }, name: Name("external-dns-a1777e88-1a9c-49b1-bca1-ec6c5e1991fc"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 183, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
96642025-03-21T02:14:27.628ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96652025-03-21T02:14:27.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96662025-03-21T02:14:27.629ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96672025-03-21T02:14:27.631ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
96682025-03-21T02:14:27.631ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96692025-03-21T02:14:27.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96702025-03-21T02:14:27.633ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
96712025-03-21T02:14:27.633ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 5a5beddc-544e-4e62-bf20-c61d9dde2785, kind: Service { id: bfbad16c-d6a4-4742-9db2-139d724c0771 }, name: Name("external-dns-bfbad16c-d6a4-4742-9db2-139d724c0771"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 221])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bfb6f529-ad0a-4d8f-8e27-90ee0b561301, kind: Service { id: a1777e88-1a9c-49b1-bca1-ec6c5e1991fc }, name: Name("external-dns-a1777e88-1a9c-49b1-bca1-ec6c5e1991fc"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 183, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 34365a8b-47e5-4623-9ff4-fa54efca49f0, kind: Service { id: 6021c9d9-927c-44a6-8331-6ab48efd9c00 }, name: Name("nexus-6021c9d9-927c-44a6-8331-6ab48efd9c00"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 210])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8d8a5138-cad9-4d17-aa4a-561116e6b6ee, kind: Service { id: e05eb799-8cf0-4702-bd3b-9994ae86118a }, name: Name("nexus-e05eb799-8cf0-4702-bd3b-9994ae86118a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 112])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c4619e25-4546-4262-8518-a5e7bdb39205, kind: Service { id: 4b42f969-b761-4b99-9019-b41841cb0f60 }, name: Name("nexus-4b42f969-b761-4b99-9019-b41841cb0f60"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 128])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
96722025-03-21T02:14:27.633ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
96732025-03-21T02:14:27.689ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: ad5b28f8-6cad-4221-97f9-e84a46a6381b, time_created: 2025-03-21T02:04:28.969535Z, time_modified: 2025-03-21T02:04:28.969535Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, is_scrimlet: true, serial_number: "buskin", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(56094621696)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
96742025-03-21T02:14:27.689ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
96752025-03-21T02:14:27.689ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96762025-03-21T02:14:27.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96772025-03-21T02:14:27.691ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96782025-03-21T02:14:27.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96792025-03-21T02:14:27.693ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96802025-03-21T02:14:27.696ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
96812025-03-21T02:14:27.706ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "82d7aade-57d1-4f5b-bce9-6a4686e5efa6", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
96822025-03-21T02:14:27.706ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
96832025-03-21T02:14:27.706ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
96842025-03-21T02:14:27.707ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "5add2e16-78ec-4c4d-8c6e-22507768abda", "content-length": "682", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
96852025-03-21T02:14:27.707ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96862025-03-21T02:14:27.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
96872025-03-21T02:14:27.711ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96882025-03-21T02:14:27.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
96892025-03-21T02:14:27.714ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96902025-03-21T02:14:27.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
96912025-03-21T02:14:27.717ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96922025-03-21T02:14:27.717ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
96932025-03-21T02:14:27.724ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 801.950941ms iteration = 21 status = {}
96942025-03-21T02:14:27.727ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
96952025-03-21T02:14:27.728ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "00248c4a-ab88-4741-b812-8c610feaff02", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
96962025-03-21T02:14:27.730ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
96972025-03-21T02:14:27.730ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 807.581739ms
96982025-03-21T02:14:27.730ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 807.59252ms iteration = 3 status = {}
96992025-03-21T02:14:27.877ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2baada4c-66ae-4bde-827d-3b7f5bcb5b03", "content-length": "31976", "date": "Fri, 21 Mar 2025 02:14:27 GMT"} })
97002025-03-21T02:14:27.901ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97012025-03-21T02:14:27.901ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97022025-03-21T02:14:27.901ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
97032025-03-21T02:14:27.905ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97042025-03-21T02:14:27.906ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97052025-03-21T02:14:27.907ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97062025-03-21T02:14:27.909ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97072025-03-21T02:14:27.909ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97082025-03-21T02:14:27.909ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97092025-03-21T02:14:28.109ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 5b18679e-aad8-42e9-9ee5-69357b37de79 file = nexus/db-queries/src/db/datastore/inventory.rs:1010
97102025-03-21T02:14:28.109ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 5b18679e-aad8-42e9-9ee5-69357b37de79 time_started = 2025-03-21 02:14:27.274665 UTC
97112025-03-21T02:14:28.109ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.186852062s iteration = 2 status = {"collection_id":"5b18679e-aad8-42e9-9ee5-69357b37de79","time_done":"2025-03-21 02:14:27.903918 UTC","time_started":"2025-03-21 02:14:27.274665 UTC"}
97122025-03-21T02:14:28.109ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 5 reason = Dependency
97132025-03-21T02:14:28.109ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 23 reason = Dependency
97142025-03-21T02:14:28.109ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
97152025-03-21T02:14:28.109ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
97162025-03-21T02:14:28.109ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
97172025-03-21T02:14:28.109ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
97182025-03-21T02:14:28.111ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97192025-03-21T02:14:28.111ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97202025-03-21T02:14:28.113ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
97212025-03-21T02:14:28.114ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
97222025-03-21T02:14:28.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
97232025-03-21T02:14:28.116ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
97242025-03-21T02:14:28.116ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
97252025-03-21T02:14:28.116ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
97262025-03-21T02:14:28.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
97272025-03-21T02:14:28.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
97282025-03-21T02:14:28.118ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97292025-03-21T02:14:28.118ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
97302025-03-21T02:14:28.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
97312025-03-21T02:14:28.120ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
97322025-03-21T02:14:28.120ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
97332025-03-21T02:14:28.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
97342025-03-21T02:14:28.143ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
97352025-03-21T02:14:28.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
97362025-03-21T02:14:28.145ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97372025-03-21T02:14:28.147ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
97382025-03-21T02:14:28.147ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
97392025-03-21T02:14:28.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
97402025-03-21T02:14:28.148ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97412025-03-21T02:14:28.150ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
97422025-03-21T02:14:28.150ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
97432025-03-21T02:14:28.150ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
97442025-03-21T02:14:28.152ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
97452025-03-21T02:14:28.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
97462025-03-21T02:14:28.154ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97472025-03-21T02:14:28.155ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
97482025-03-21T02:14:28.155ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
97492025-03-21T02:14:28.155ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
97502025-03-21T02:14:28.156ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 47.230472ms iteration = 23 status = {"physical_disks_added":0}
97512025-03-21T02:14:28.156ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97522025-03-21T02:14:28.158ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
97532025-03-21T02:14:28.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
97542025-03-21T02:14:28.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
97552025-03-21T02:14:28.160ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
97562025-03-21T02:14:28.160ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 50.928956ms iteration = 5 status = {"blueprint_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","inventory_collection_id":"5b18679e-aad8-42e9-9ee5-69357b37de79","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
97572025-03-21T02:14:29.723ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 299 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = c4282c82-01a0-4b39-b2ff-ed8e49d837a4 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
97582025-03-21T02:14:36.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 62 reason = Timeout
97592025-03-21T02:14:36.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
97602025-03-21T02:14:36.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 62 reason = Timeout
97612025-03-21T02:14:36.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97622025-03-21T02:14:36.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
97632025-03-21T02:14:36.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97642025-03-21T02:14:36.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
97652025-03-21T02:14:36.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
97662025-03-21T02:14:36.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97672025-03-21T02:14:36.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
97682025-03-21T02:14:36.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97692025-03-21T02:14:36.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97702025-03-21T02:14:36.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97712025-03-21T02:14:36.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97722025-03-21T02:14:36.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97732025-03-21T02:14:36.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
97742025-03-21T02:14:36.933ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97752025-03-21T02:14:36.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97762025-03-21T02:14:36.943ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
97772025-03-21T02:14:36.943ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.175388ms iteration = 62 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
97782025-03-21T02:14:36.952ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 30.086274ms iteration = 62 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
97792025-03-21T02:14:39.714ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:42273
97802025-03-21T02:14:39.714ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42273 req_id = f49111ea-f356-40a8-9c15-ba8de54e4a43 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
97812025-03-21T02:14:39.714ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42273 req_id = f49111ea-f356-40a8-9c15-ba8de54e4a43 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
97822025-03-21T02:14:39.716ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2588 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42273 req_id = f49111ea-f356-40a8-9c15-ba8de54e4a43 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
97832025-03-21T02:14:39.716ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42273 req_id = 7b51c6ba-2565-4784-84d1-081a18c16b29 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
97842025-03-21T02:14:39.717ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42273 req_id = 7b51c6ba-2565-4784-84d1-081a18c16b29 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
97852025-03-21T02:14:39.719ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2362 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42273 req_id = 7b51c6ba-2565-4784-84d1-081a18c16b29 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
97862025-03-21T02:14:39.723ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 239 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 5a2cd699-17f5-4f1f-9998-8cb3da227b4e response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
97872025-03-21T02:14:46.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 63 reason = Timeout
97882025-03-21T02:14:46.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
97892025-03-21T02:14:46.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 63 reason = Timeout
97902025-03-21T02:14:46.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97912025-03-21T02:14:46.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
97922025-03-21T02:14:46.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97932025-03-21T02:14:46.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97942025-03-21T02:14:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
97952025-03-21T02:14:46.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97962025-03-21T02:14:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97972025-03-21T02:14:46.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97982025-03-21T02:14:46.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97992025-03-21T02:14:46.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98002025-03-21T02:14:46.932ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
98012025-03-21T02:14:46.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
98022025-03-21T02:14:46.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
98032025-03-21T02:14:46.942ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
98042025-03-21T02:14:46.942ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.798345ms iteration = 63 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
98052025-03-21T02:14:46.952ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
98062025-03-21T02:14:46.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
98072025-03-21T02:14:46.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.141997ms iteration = 63 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
98082025-03-21T02:14:49.722ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 258 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = cd27f85a-db31-48b5-8f42-409064a0c6e5 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
98092025-03-21T02:14:56.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
98102025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
98112025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
98122025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 22 reason = Timeout
98132025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 22 reason = Timeout
98142025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
98152025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
98162025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
98172025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
98182025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
98192025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
98202025-03-21T02:14:56.923ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
98212025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
98222025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 22 reason = Timeout
98232025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98242025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
98252025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 22 reason = Timeout
98262025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
98272025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 22 reason = Timeout
98282025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
98292025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
98302025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
98312025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
98322025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
98332025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
98342025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
98352025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
98362025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
98372025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
98382025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
98392025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
98402025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
98412025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
98422025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
98432025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
98442025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 22 reason = Timeout
98452025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
98462025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 22 reason = Timeout
98472025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
98482025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
98492025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
98502025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
98512025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
98522025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98532025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
98542025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
98552025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
98562025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
98572025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
98582025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
98592025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
98602025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
98612025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
98622025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
98632025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
98642025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
98652025-03-21T02:14:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
98662025-03-21T02:14:56.925ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
98672025-03-21T02:14:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
98682025-03-21T02:14:56.925ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98692025-03-21T02:14:56.927ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 4.398067ms iteration = 22 status = {"errors":[],"finish_invoked_ok":[]}
98702025-03-21T02:14:56.927ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.666741ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
98712025-03-21T02:14:56.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
98722025-03-21T02:14:56.927ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
98732025-03-21T02:14:56.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
98742025-03-21T02:14:56.927ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98752025-03-21T02:14:56.928ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98762025-03-21T02:14:56.929ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
98772025-03-21T02:14:56.929ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98782025-03-21T02:14:56.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98792025-03-21T02:14:56.930ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98802025-03-21T02:14:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
98812025-03-21T02:14:56.930ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
98822025-03-21T02:14:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
98832025-03-21T02:14:56.930ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1605, tv_nsec: 549773343 } })
98842025-03-21T02:14:56.930ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
98852025-03-21T02:14:56.931ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98862025-03-21T02:14:56.932ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1605, tv_nsec: 551805978 } })
98872025-03-21T02:14:56.932ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
98882025-03-21T02:14:56.932ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
98892025-03-21T02:14:56.932ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
98902025-03-21T02:14:56.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
98912025-03-21T02:14:56.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
98922025-03-21T02:14:56.933ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
98932025-03-21T02:14:56.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
98942025-03-21T02:14:56.935ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98952025-03-21T02:14:56.938ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
98962025-03-21T02:14:56.938ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
98972025-03-21T02:14:56.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
98982025-03-21T02:14:56.940ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98992025-03-21T02:14:56.941ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
99002025-03-21T02:14:56.941ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1605, tv_nsec: 561268143 } })
99012025-03-21T02:14:56.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
99022025-03-21T02:14:56.942ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
99032025-03-21T02:14:56.942ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
99042025-03-21T02:14:56.942ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
99052025-03-21T02:14:56.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
99062025-03-21T02:14:56.942ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
99072025-03-21T02:14:56.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
99082025-03-21T02:14:56.946ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
99092025-03-21T02:14:56.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
99102025-03-21T02:14:56.967ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 44.408274ms iteration = 22 status = {"errors":[],"requests_created_ok":[]}
99112025-03-21T02:14:56.969ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
99122025-03-21T02:14:56.969ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
99132025-03-21T02:14:56.969ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
99142025-03-21T02:14:56.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
99152025-03-21T02:14:56.972ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 48.931241ms iteration = 22 status = {"errors":[],"garbage_collect_requested":[]}
99162025-03-21T02:14:56.972ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
99172025-03-21T02:14:56.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
99182025-03-21T02:14:56.973ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
99192025-03-21T02:14:56.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99202025-03-21T02:14:56.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
99212025-03-21T02:14:56.974ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
99222025-03-21T02:14:56.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
99232025-03-21T02:14:56.974ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
99242025-03-21T02:14:56.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
99252025-03-21T02:14:56.975ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
99262025-03-21T02:14:56.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
99272025-03-21T02:14:56.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
99282025-03-21T02:14:56.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99292025-03-21T02:14:56.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99302025-03-21T02:14:56.977ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99312025-03-21T02:14:56.979ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99322025-03-21T02:14:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99332025-03-21T02:14:56.979ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 56.87043ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
99342025-03-21T02:14:56.980ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99352025-03-21T02:14:56.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
99362025-03-21T02:14:56.981ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
99372025-03-21T02:14:56.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
99382025-03-21T02:14:56.982ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
99392025-03-21T02:14:56.982ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
99402025-03-21T02:14:56.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
99412025-03-21T02:14:56.982ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
99422025-03-21T02:14:56.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
99432025-03-21T02:14:56.983ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 60.599365ms iteration = 22 status = 0
99442025-03-21T02:14:56.984ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99452025-03-21T02:14:56.985ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
99462025-03-21T02:14:56.985ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
99472025-03-21T02:14:56.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
99482025-03-21T02:14:56.987ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
99492025-03-21T02:14:56.987ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 64.102856ms iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
99502025-03-21T02:14:57.014ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 90.937303ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
99512025-03-21T02:14:57.015ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
99522025-03-21T02:14:57.015ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 92.391659ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
99532025-03-21T02:14:57.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
99542025-03-21T02:14:57.017ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
99552025-03-21T02:14:57.017ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
99562025-03-21T02:14:57.017ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
99572025-03-21T02:14:57.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
99582025-03-21T02:14:57.020ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99592025-03-21T02:14:57.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
99602025-03-21T02:14:57.024ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99612025-03-21T02:14:57.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99622025-03-21T02:14:57.027ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
99632025-03-21T02:14:57.027ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
99642025-03-21T02:14:57.027ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d16f9e1-876d-4673-91e2-0c8abe7e917f", "content-length": "26", "date": "Fri, 21 Mar 2025 02:14:57 GMT"} })
99652025-03-21T02:14:57.027ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
99662025-03-21T02:14:57.029ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
99672025-03-21T02:14:57.033ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
99682025-03-21T02:14:57.033ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 110.765468ms iteration = 22 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
99692025-03-21T02:14:57.034ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 110.935192ms iteration = 24 status = {"physical_disks_added":0}
99702025-03-21T02:14:57.034ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
99712025-03-21T02:14:57.034ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 111.414639ms iteration = 64 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
99722025-03-21T02:14:57.053ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
99732025-03-21T02:14:57.054ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
99742025-03-21T02:14:57.054ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 130.921249ms iteration = 22 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
99752025-03-21T02:14:57.059ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 136.7107ms iteration = 22 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
99762025-03-21T02:14:57.070ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
99772025-03-21T02:14:57.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
99782025-03-21T02:14:57.072ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99792025-03-21T02:14:57.074ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
99802025-03-21T02:14:57.074ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
99812025-03-21T02:14:57.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
99822025-03-21T02:14:57.075ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99832025-03-21T02:14:57.077ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
99842025-03-21T02:14:57.077ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
99852025-03-21T02:14:57.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
99862025-03-21T02:14:57.097ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
99872025-03-21T02:14:57.097ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
99882025-03-21T02:14:57.097ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ceaa8e8-b8b7-4f75-85ee-e27b1edad3a6", "content-length": "26", "date": "Fri, 21 Mar 2025 02:14:57 GMT"} })
99892025-03-21T02:14:57.097ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
99902025-03-21T02:14:57.099ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
99912025-03-21T02:14:57.131ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
99922025-03-21T02:14:57.131ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
99932025-03-21T02:14:57.131ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "987b8e79-e990-438f-89ac-e82a65ec9b7e", "content-length": "26", "date": "Fri, 21 Mar 2025 02:14:57 GMT"} })
99942025-03-21T02:14:57.131ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
99952025-03-21T02:14:57.133ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
99962025-03-21T02:14:57.148ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
99972025-03-21T02:14:57.148ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
99982025-03-21T02:14:57.161ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8dc9fdd-1039-48af-854d-91b34bbeeaab", "content-length": "372", "date": "Fri, 21 Mar 2025 02:14:57 GMT"} })
99992025-03-21T02:14:57.161ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
100002025-03-21T02:14:57.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
100012025-03-21T02:14:57.207ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
100022025-03-21T02:14:57.207ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd2d8fe0-bb6c-4004-a227-4ae185e32498", "content-length": "691", "date": "Fri, 21 Mar 2025 02:14:56 GMT"} })
100032025-03-21T02:14:57.208ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
100042025-03-21T02:14:57.208ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
100052025-03-21T02:14:57.208ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
100062025-03-21T02:14:57.209ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 286.042929ms iteration = 22 status = {}
100072025-03-21T02:14:57.212ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
100082025-03-21T02:14:57.212ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
100092025-03-21T02:14:57.212ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "ff9e64e7-9d41-455a-a0a1-1425d25da660", "date": "Fri, 21 Mar 2025 02:14:56 GMT"} })
100102025-03-21T02:14:57.212ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
100112025-03-21T02:14:57.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
100122025-03-21T02:14:57.213ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 290.393146ms iteration = 22 status = {}
100132025-03-21T02:14:57.215ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
100142025-03-21T02:14:57.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
100152025-03-21T02:14:57.218ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
100162025-03-21T02:14:57.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
100172025-03-21T02:14:57.221ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
100182025-03-21T02:14:57.221ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
100192025-03-21T02:14:57.221ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "181f82a9-d460-4825-a67a-77d1b1f5f94c", "date": "Fri, 21 Mar 2025 02:14:56 GMT"} })
100202025-03-21T02:14:57.223ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 300.40705ms iteration = 22 status = {}
100212025-03-21T02:14:57.228ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
100222025-03-21T02:14:57.229ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7c54bc8-0e5a-48d7-95e2-bc1e518b8c9a", "content-length": "1", "date": "Fri, 21 Mar 2025 02:14:56 GMT"} })
100232025-03-21T02:14:57.231ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
100242025-03-21T02:14:57.231ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
100252025-03-21T02:14:57.233ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 310.535766ms iteration = 22 status = null
100262025-03-21T02:14:57.271ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100272025-03-21T02:14:57.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100282025-03-21T02:14:57.273ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
100292025-03-21T02:14:57.273ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100302025-03-21T02:14:57.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100312025-03-21T02:14:57.283ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
100322025-03-21T02:14:57.283ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a77c0a2-9ca2-4e2e-852f-053479e88e48", "content-length": "134", "date": "Fri, 21 Mar 2025 02:14:56 GMT"} })
100332025-03-21T02:14:57.283ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
100342025-03-21T02:14:57.283ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
100352025-03-21T02:14:57.283ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
100362025-03-21T02:14:57.283ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
100372025-03-21T02:14:57.283ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
100382025-03-21T02:14:57.283ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
100392025-03-21T02:14:57.283ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
100402025-03-21T02:14:57.283ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
100412025-03-21T02:14:57.283ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
100422025-03-21T02:14:57.283ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
100432025-03-21T02:14:57.283ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
100442025-03-21T02:14:57.283ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100452025-03-21T02:14:57.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100462025-03-21T02:14:57.286ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
100472025-03-21T02:14:57.286ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b7c37a6-a047-4019-b4a9-917e9bf88f0f", "content-length": "2", "date": "Fri, 21 Mar 2025 02:14:56 GMT"} })
100482025-03-21T02:14:57.286ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
100492025-03-21T02:14:57.286ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6021836-13d8-4d0b-bca9-aa9b265fd0e0", "content-length": "52", "date": "Fri, 21 Mar 2025 02:14:56 GMT"} })
100502025-03-21T02:14:57.286ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
100512025-03-21T02:14:57.286ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
100522025-03-21T02:14:57.383ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "6bfbe26d-a2ec-46d3-8743-62c41a5381af", "date": "Fri, 21 Mar 2025 02:14:56 GMT"} })
100532025-03-21T02:14:57.383ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
100542025-03-21T02:14:57.383ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
100552025-03-21T02:14:57.383ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "88b6d21a-a6ae-499f-beb0-4d497f0fd9ed", "content-length": "682", "date": "Fri, 21 Mar 2025 02:14:56 GMT"} })
100562025-03-21T02:14:57.383ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100572025-03-21T02:14:57.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100582025-03-21T02:14:57.387ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100592025-03-21T02:14:57.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100602025-03-21T02:14:57.390ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100612025-03-21T02:14:57.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100622025-03-21T02:14:57.392ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100632025-03-21T02:14:57.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100642025-03-21T02:14:57.399ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 476.734608ms iteration = 22 status = {}
100652025-03-21T02:14:59.723ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 259 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 99c3ffdd-72d8-4040-adf6-17c94189e1ae response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a